I do not know if this issue is specific to Oracle/WFS, that is the only place that I have tested this.
When calling WFS, eg, GetFeature, the GML response contains illegal XML characters that exist in the source data. This can then cause issues in the client. In our case the client is OpenLayers and a browser. Firefox throws an error and doesn't process the data, this is better behaviour than Chrome/Edge which silently truncate the GML leading to missing features without any clear indication why.
Illegal XML characters: /[\x00-\x08\x0B\x0C\x0E-\x1F\x7F-\x9F]/
Copilot suggests fixing this issue here:
|
if (encoded_value == NULL) { |
however I do not have the knowledge of the codebase to confidently implement this fix myself.
Our fix in the client:
const sanitizeXmlData = (data: string): string => {
// Our app stores characters *?=><!&|; as control characters 16-23,27
const map: Record<string, string> = {
"\x10": "*",
"\x11": "?",
"\x12": "=",
"\x13": ">",
"\x14": "<",
"\x15": "!",
"\x16": "&",
"\x17": "|",
"\x1B": ";",
};
// Match all illegal control chars and replace with mapped value, or empty string if not mapped
return data.replace(/[\x00-\x08\x0B\x0C\x0E-\x1F\x7F-\x9F]/g, (ch) => map[ch] ?? "");
};
const data = await fireRequest('GetFeature', requestXml, params.logContext, params.abortController)
let { numberMatched, numberReturned } = extractNumberAttributes(data)
let olFeatures = getFeaturesFromXml(data)
// MapServer does not filter out illegal XML characters from its GML response, leading to Firefox throwing an error and Chrome and Edge
// silently truncating the response at the illegal character, leading to missing features which is only spotted by people who know they are
// there/are specifically looking for them.
if (olFeatures.length !== numberReturned) {
const responseLogContext = { ...logContext, responseData: data }
logger.warn(LOG_SRC, `Number of features returned (${olFeatures.length}) does not match numberReturned in response (${numberReturned}) for ${options.name} ${params.logContext}`, responseLogContext)
const sanitizedData = sanitizeXmlData(data)
if (sanitizedData !== data) {
logger.error(LOG_SRC, `Response contains illegal XML characters which may have caused parsing issues`, responseLogContext)
// Re-set variables from sanitised data. In practice we only need to re-read numberMatched and numberReturned for Firefox, Chrome/Edge parse up to the illegal character
; ({ numberMatched, numberReturned } = extractNumberAttributes(sanitizedData))
olFeatures = getFeaturesFromXml(sanitizedData)
I do not know if this issue is specific to Oracle/WFS, that is the only place that I have tested this.
When calling WFS, eg, GetFeature, the GML response contains illegal XML characters that exist in the source data. This can then cause issues in the client. In our case the client is OpenLayers and a browser. Firefox throws an error and doesn't process the data, this is better behaviour than Chrome/Edge which silently truncate the GML leading to missing features without any clear indication why.
Illegal XML characters: /[\x00-\x08\x0B\x0C\x0E-\x1F\x7F-\x9F]/
Copilot suggests fixing this issue here:
MapServer/src/mapgml.c
Line 1223 in 320a725
Our fix in the client:
const sanitizeXmlData = (data: string): string => { // Our app stores characters *?=><!&|; as control characters 16-23,27 const map: Record<string, string> = { "\x10": "*", "\x11": "?", "\x12": "=", "\x13": ">", "\x14": "<", "\x15": "!", "\x16": "&", "\x17": "|", "\x1B": ";", }; // Match all illegal control chars and replace with mapped value, or empty string if not mapped return data.replace(/[\x00-\x08\x0B\x0C\x0E-\x1F\x7F-\x9F]/g, (ch) => map[ch] ?? ""); }; const data = await fireRequest('GetFeature', requestXml, params.logContext, params.abortController) let { numberMatched, numberReturned } = extractNumberAttributes(data) let olFeatures = getFeaturesFromXml(data) // MapServer does not filter out illegal XML characters from its GML response, leading to Firefox throwing an error and Chrome and Edge // silently truncating the response at the illegal character, leading to missing features which is only spotted by people who know they are // there/are specifically looking for them. if (olFeatures.length !== numberReturned) { const responseLogContext = { ...logContext, responseData: data } logger.warn(LOG_SRC, `Number of features returned (${olFeatures.length}) does not match numberReturned in response (${numberReturned}) for ${options.name} ${params.logContext}`, responseLogContext) const sanitizedData = sanitizeXmlData(data) if (sanitizedData !== data) { logger.error(LOG_SRC, `Response contains illegal XML characters which may have caused parsing issues`, responseLogContext) // Re-set variables from sanitised data. In practice we only need to re-read numberMatched and numberReturned for Firefox, Chrome/Edge parse up to the illegal character ; ({ numberMatched, numberReturned } = extractNumberAttributes(sanitizedData)) olFeatures = getFeaturesFromXml(sanitizedData)