B-C-14E1DF3

Monitor and reduce 5xx non-origin errors

Blog Post created by B-C-14E1DF3 Employee on Aug 3, 2015

Problem:

 

The customer noticed unusual high percentage of non-client origin 500 and 502 errors. The non-client origin in the architecture involved SiteSpect cloud and Akamai ESI.

 

There were two main issues client wanted addressed by Akamai

  • Provide client the visibility to isolate the source of 500 and 502 errors.
  • Log and report 500 and 502 errors in a manner to isolate the percentage of SiteSpect versus Akamai ESI versus client origin errors.

 

 

The architecture involved end user request intercepted at Akamai is first checked using IPA calls to client origin and determined whether to pass through SIteSpect. If the SiteSpect is enabled, the request is routed to SiteSpect cloud, which constitutes LEG 1 portion of the request. If the LEG 1 response code is good then Akamai makes a request to client origin, which constitutes the LEG 2 portion of the request.  If the LEG 2 response from the origin is good, Akamai performs ESI processing and finally end user response is sent.

 

Implementation:

 

Akamai put forward the following solution to fix logging and failover scenarios.

 

  • Enhanced Logging with separate co codes to capture failover scenarios

Default cp code for the configuration is num1

If the request goes through SiteSpect then,

LEG 1 cp code is num2

LEG 2 cp code is num3

Error page on LEG 1 cp code is num4

Error page on LEG 2 cp code num5

 

  • Handle all possible failover scenarios
    • Client origin 5xx response codes - Akamai does not failover, client origin serves custom error page along with custom header. Akamai uses custom header to determine not to failover and serve Akamai custom error page.
    • SiteSpect 5xx response codes - Akamai serves custom error page from NetStorage with LEG 1 response code
    • Akamai ESI 5xx response codes - Akamai serves custom error page from NetStorage with LEG 2 response code. The LEG 2 response code could potentially be from client origin and/or Akamai ESI.

 

 

 

Metadata to handle Failover 500 and 502

 

<match:uri.wildcard value="/akamai_error_download.html" result="false">

                    <match:response.header name="X-CUSTOM-Origin-Error-Page" name-wildcard="off" result="false">

<match:response.header name="leg2-failover-page" name-wildcard="off" result="false">

<match:response.status value="500" result="true">

<forward:availability.fail-action2>

<status>on</status>

<type>serve-alternate</type>

<request-host>download.example.com</request-host>

<path>/akamai_error_download.html?error-code=500&amp;leg=%(LEG)</path>

</forward:availability.fail-action2>

</match:response.status>

<match:response.status value="502" result="true">

<forward:availability.fail-action2>

<status>on</status>

<type>serve-alternate</type>

<request-host>download.example.com</request-host>

                                    <path>/akamai_error_download.html?error-code=502&amp;leg=%(LEG)</path>

</forward:availability.fail-action2>

</match:response.status>

</match:response.header>

</match:response.header>

</match:uri.wildcard>

 

METADATA TO AVOID ESI 502 ERRORS

 

We noticed very high ESI processing 502 errors due bug (CR#: 581293 ;Depends On EdgeSuite CR#: 581257) in Akamai ghost. The metadata fix below was applied to resolve the ESI 502 errors.

 

<edgeservices:lma.rollback-transmit-filter-failure>on</edgeservices:lma.rollback-transmit-filter-failure>

 

 

 

Metadata to Log LEG 1 and LEG 2

 

<match:request.header name="SSVisit">

                    <match:request.type value="CLIENT_REQ">

                        <edgeservices:modify-outgoing-response.add-header>

<status>on</status>

<name>X-Akamai-Request-ID2</name>

<value>%(AK_GHOST_SERVICE_IP):%(AK_REQUEST_ID)</value>

                        </edgeservices:modify-outgoing-response.add-header>

                    </match:request.type>

<assign:extract-value>

<location>Client_Request_Header</location>

<location-id>AkaIPReqIdToSS</location-id>

<variable-name>AKAIPREQID</variable-name>

</assign:extract-value>

</match:request.header>

<match:metadata-stage value="client-done">

                    <reporting:lds.custom-field>%(X_FORWARDED_FOR);%(ACCEPT_LANGUAGE);%(SSVISIT);%(AKAIPREQID);%(SSLB_COOKIE);ss-%(SITESPECT);X-Akamai-Request-ID2=%(X_AKAMAI_REQUEST_ID2)</reporting:lds.custom-field>

</match:metadata-stage>

<match:request.header name="SiteSpect" result="true">

                    <match:request.header name="SSVisit" result="true">

<reporting:cpcode>259722</reporting:cpcode>

                        <assign:variable>

<name>LEG</name>

<value>2</value>

</assign:variable>

</match:request.header>

</match:request.header>

<comment:note value="End Feature advanced"/>

</match:variable>

 

 

 

200 response code and Request ID2 logged:

 

1 Record tag          f

2 Start time 1430804380.910 (Tue May 5 05:39:40 2015 GMT)

10 Forward IP          74.217.214.115

11 Client IP 127.0.0.1

12 HTTP method    GET

13 HTTP status code 200

31 Request id 4f646270.4f64e18f

47 Custom Field ;;;;;ss-TRUE;X-Akamai-Request-ID2=184.25.56.150:5ce33a9f

 

 

502 response code terminated at LEG 1 and no RequestID2 logging (implied no transaction at LEG 2):

 

1 Record tag          f

2 Start time 1430804387.959 (Tue May 5 05:39:47 2015 GMT)

10 Forward IP          74.217.214.115

11 Client IP 127.0.0.1

12 HTTP method    GET

14 HTTP status code 502

31 Request id          4f64f291

47 Custom Field      ;;;;;ss-;X-Akamai-Request-ID2=

 

LEG 1 and LEG 2 and 502 response code from CLIENT_ORIGIN:

 

LEG 1:

1 Record tag f

2 Start time 1430920666.643 (Wed May 6 13:57:46 2015 GMT)

10 Forward IP 69.25.155.7

12 HTTP method GET

14 HTTP status code         502

31 Request id 25cdc81.25cdd4c

47 Custom Field ;;;;;ss-TRUE;X-Akamai-Request-ID2=23.74.9.181:4a134bc

 

LEG 2:

1 Record tag f

2 Start time 1430920666.703 (Wed May 6 13:57:46 2015 GMT)

10 Forward IP 64.30.224.58

12 HTTP method GET

13 HTTP status code         502

31 Request id d397a60.4a134bc

47 Custom Field ;;true;207.86.164.92:25cdc81;;ss-FALSE;X-Akamai-Request-ID2=

 

 

The following omniture report shows the real time SiteSpect/ Akamai and client origin 5xx errors. The data from the graph is captured from the tracking code placed on custom and Akamai custom error pages.

 

 

GRAPH 1: With SITESPECT_ORIGIN and ESI

 

9997 – LEG 1 and LEG 2 5XX

9999 – CLIENT_ORIGIN 5XX

 

Graph1 is the initial state without the Akamai implementation in place.

 

 

WithoutAkamaiImplementation.png

 

 

 

 

GRAPH 2: With SITESPECT_ORIGIN Pass Through

 

Graph 2 shows when SiteSpect is pass through the 5xx errors come down drastically.  The graph determined SiteSpect to take corrective steps bringing down 5xx errors on LEG 1.

 

ByPassSiteSpect.png

 

 

GRAPH 3: With SITESPECT_ORIGIN and ESI and ESI workaround fix

 

Graph 3 shows SiteSpect and ESI errors are managed. The dip to the right is the ideal state where LEG 1 (SiteSpect) and LEG 2(Origin + Akamai ESI) errors are minimized.

 

AfterFixestoAkamaiESIandSiteSpect.png

Outcomes