S3 Beacon Upload Logs

Document created by Dave Murphy Employee on Jul 20, 2017Last modified by Sheril Joseph on Aug 11, 2017
Version 2Show Document
  • View in full screen mode

SOASTA mPulse provides beacon data via Amazon S3 logs. The logged data is derived from the raw data HTTP access logs, which is passed through several filters and transformers to arrive at a somewhat processed state that, if configured, is then stored in S3 via the Configure App, Beacons tab.

Note: If you haven't configured Raw Beacon Upload to your own Amazon EC2, S3 Account, you can do so now. Refer to the Beacon Settings, Upload Setup for Domain-Specific Beacon Logs for steps.

Users should note that the actual JSON-format beacon elements found in a given bucket will vary, including some fields not found here. The contents of your S3 store depend on the makeup of the domain for a given beacon, the site content, and the browser in use, as well as whether custom timers and custom metrics are configured for your account.

Working with S3 Buckets

The following steps describe accessing the S3 Data via the Amazon UI. Users may opt to use other methods. For example, by using the command line interface provided by the `s3cmd` command, which is available for all operating systems here.

  1. Login to the Amazon EC2 Account where your beacon data is stored. The root bucket folder's name is the same as the configured app's API Key.

  1. Select and open the root bucket folder. The folder whose name is also your mPulse API Key (obfuscated below).

  1. The Dates folders are listed and the breadcrumb at the top of the page shows the current path. Select and open the desired date folder.
  2. In the Date folder, one folder for each IP address from which Beacons are collected is listed. Open this folder and the gzip logs specific to that IP address are shown.

  1. Select and download the desired gzip file.
  2. Once downloaded, open the Beacon log in your preferred editor. A "line" is defined by left- and right-curly-braces standard in JSON syntax.

TIP:  You can use a JSON formatter, such as this JSON Formatter & Validator in order to better understand the beacon log structure (shown below).

The S3 log (shown above in the JSON formatter) includes the Base Request Objects (shown in Table 1 below), while the sub-group nodes include user_agent, geo, params, headers, session, and custom_metrics.

The remainder of this article presents details about many common objects found in this log's base request, and subgroups. The actual beacon data will vary for any given web app.

Note: The following tables use the notation EPOCH as shorthand for Unix EPOCH time. Refer to Unix time for more information.

 

 

Base Request Objects

The Base Request objects are first-level objects (e.g. non-indented in the JSON-format Beacon Log) and are found at the top of the Beacon log.

Table 1: Base Request Objects

ElementDescription
compression_typesThe HTTP compression types supported by the user's browser, if any. In this example, null means no compression mode was in use.
domainThe mPulse domain configured for the beacon. For example, mydomain.com.
http_methodThe HTTP method (i.e. GET, PUT, POST, etc.).
http_versionThe HTTP version in use. For example, HTTP/1.1.
http_referrerThe URL of the page that sent the beacon. This value should match the url parameter, if not, a warning will also be printed.
keyThe alphanumeric mPulse domain API key in use. For example, R6SH7-84RFL-GQQ8S-CW6MF-W5RWR (not a real key).
mobile_connection_typeThe type of network connection on use in a mobile device. This has the following values: null, Ethernet, WiFi, 2G, 3G, 4G, LTE. This element is often not present since most browsers have stopped supporting it.
proxy_addressThe address of the CloudFlare network proxy host in use. This will always be one of CloudFlare's IP addresses.
page_groupThe name of the mPulse Page Group in use. For example, www_home.
remote_ipThe public IP address of the remote visitor (e.g. from which the request was initiated).
timestampThe EPOCH timestamp for the given beacon in milliseconds. For example, 1404864000314.
urlThe URL of the page from which the beacon originated. For example, http://www.mydomain.com

 

SubGroup Objects

The sub-group objects are additional second-level objects (e.g. indented in the JSON-format Beacon Log) and are found after the list of Base Request objects.

 

Table 2: User Agent Objects

The following fields are commonly found in the user_agent sub-group.

ElementDescription
familyThe user agent family. For example, Chrome, IE, Firefox, Safari, etc.
majorThe major release to which the user agent belongs. For example, IE 11.
minorThe minor, or point release of the user agent. For example, 0.
osThe Operating System of the user agent. For example, Windows, Mac OS X, etc.
osversionThe version of the OS already listed. For example, 7 (e.g. as in Windows 7).
mobileA Boolean value to indicate a mobile client. 0 for non-mobile, 1 for mobile.
rawThe full or raw User Agent String, such as found in an HTTP Header. For example, Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko.
modelThe desktop or mobile device model. For example, iPhone or an Android device such as ME301T (as in an Asus ME30ITmobile device).

 

Table 3: Geographic Objects

The following fields are commonly found in the geo sub-group. As noted in the introductory section above, Geo data is derived by the underlying mPulse database from the `remote_ip` address of the customer.

ElementDescription
ccThe ISO codes for US & Canada, and FIPS codes for all other countries.
cityThe nearest city. For example, London, Astoria, Elk Grove, etc. This value can also be null
latThe lattitude. For example, 38.441193.
lonThe longitude. For example, -121.3064. The combined lat and lon together fix the beacon location.
netspeedThe network speed of the client. For example, Cable/DSL, Cellular, etc. This value can also be null.
orgThe ISP or organization of the client user. For example, Frontier Communications, Time Warner Cable, or any other ISP.
rgThe region within the given country (cc). In the S3 logs, mPulse writes ISO codes for US & Canada, and FIPS codes for all other countries.

 

Table 4: Timers

The following fields are commonly found in the timers sub-group. This sub-group can contain both out-of-the-box mPulse timers (e.g. navigation, resource, and performance timers), as well as custom timers in use in the domain when the beacon data was captured. mPulse measures actual times reported by the browser. The rows in this table are shown

ElementDescription
t_doneThe full page load time, which starts when the user shows intent of loading a page, and ends when the page has completed loading.
t_pageThe time from the first byte to onload
t_respThe time from the request start to first byte.  t_resp + t_page will always == t_done.
t_otherA comma-separated list of additional timers set by page developer. Each timer is of the format name|value
t_loadIf the page was prerendered, this is the time to fetch and prerender the page.
t_prerenderedIf the page was prerendered, this is the time from start of prefetch to the actual page display. It may only be useful for debugging.
t_postrenderIf the page was prerendered, this is the time from prerender finish to actual page display. It may only be useful for debugging.
boomerangThe time taken to load up and execute the Boomerang.
boomr_fbThe time taken from page request to boomerang load.
boomr_latThe network latency for boomerang (request to first byte).
boomr_ldThe time taken from the boomerang loader start to the boomerang on the page.
customCustom timers are included as a series in the order they were created in the user account. For example, custom0, custom1, custom2, etc.
dnsThe DNS lookup.
domLoadLoad time of the Document Object Model (DOM) of the page expressed in milliseconds (ms). Maps onto an attribute from the browser's NavigationTiming API's window.performance.timing.domLoading.
domReadyLoad time until domReady. domReady is used to ensure that DOM elements exist when the code that attempts to access them is executed by placing that code within the 'domready' event.
sslThe Secure Socket Layer (SSL) handshake time. This field appears only if SSL is in use.
mobile_connection_typeThe type of network connection in use in a mobile device. This has the following values: null, Ethernet, WiFi, 2G, 3G, 4G, LTE. This element is highly likely to not be present since all browsers that did support it in the past have now stopped supporting it.
renderStart

The load time until renderStart (e.g. until the page is rendered by the user agent) expressed in millisecond (ms).

t_configfbThe time to first byte of the config.js request.
t_configjsThe time to last byte of config.js request.
t_domloadedThe time for the DOM to complete loading.
tcpThe time taken for the TCP connect.

 

Table 5: Headers (Most Commonly Used)

The following fields are commonly found in the headers sub-group. This sub-group can contain HTTP headers as well as headers from CDNs and other services. Many additional HTTP headers, including proprietary ones, may appear in your S3 log.

ElementDescription
accept-languageThe HTTP accept-language field defined by the user's browser.
acceptThis HTTP header field is used to specify certain media types which are acceptable for the response.According to the W3, accept headers can be used to indicate that the request is specifically limited to a small set of desired types, as in the case of a request for an in-line image. Refer to Header Field Definitions.
accept-encodingThe type of compression supported by the browser.
cache-controlThis HTTP header is used to enforce caching directives along the request-response chain.
cf-rayCloud Flare-specific header used for internal debugging. This field should be ignored.
cf-visitorCloud Flare-specific header used for internal debugging. This field should be ignored.
cf-connecting-ipCloud Flare-specific header used for internal debugging. This field should be ignored.
connectionThe HTTP Connection Status. For example, Keep-Alive.
content-lengthThis HTTP header detects the length of the beacon; for internal use only.
content-typeThis HTTP header detects the type of the beacon; for internal use only.
hostThe mPulse Collector in use.
true-client-ipCloud Flare-specific header that is not used by mPulse.
x-forwarded-forThis HTTP header field is used to identify the originating IP address of a client connecting  to a server via a proxy or load balancer..
x-forwarded-protoThis header is used to determine the original protocol in use in a request.

 

Table 6: Common Params

The following fields are commonly found in the params sub-group.

ElementDescription
domainThe mPulse domain (or app) configured for the beacon.
dom.imgNumber of Images in the DOM.
dom.lnNumber of Nodes in the DOM.
dom.resNumber of ResourceTiming resources in the DOM.
dom.scriptNumber of Script Files in the DOM.
dom.szSize of data in the DOM.
timestampThe EPOCH timestamp for the given beacon. For example, 1404864000314.
remote_ipThe IP address of the remove server. For example, 192.168.12.
keyThe alphanumeric mPulse domain key in use. For example, R6SH7-84RFL-GQQ8S-CW6MF-W5RWR (not a real key).
http_methodThe HTTP method (i.e. GET, PUT, POST, etc.).
http_versionThe HTTP version in use. For example, HTTP/1.1.
http_referrerThe HTTP header field idnetifying the URL that linked to the resource being requested. In this example, null means no referrer was in use.
compression_typesThe HTTP compression type in use on the originating server, if any. In this example, null means no compression mode was in use.
proxy_addressThe address of the network proxy host in use, if any. Otherwise, null.
mem.totalTotal memory (Chrome only).
mem.usedMemory used in bytes (Chrome only).
mobile_connection_typeThe type of network connection on use in a mobile device. For example, WiFi. Other known values for this type include: unk, [NULL], 3G, c2g, none, c3g, c4g, Ethernet, WiFi, 2G, and 4G
page_groupThe name of the mPulse Page Group in use. For example, www_home.
rThe URL of page that set the start time of the beacon (e.g. Referrer from the last navigation).
r2The URL of the referrer of the current page (e.g. Referrer from document.referrer). Only set if different from r and strict_referrer has been explicitly turned off.
uThe full URL of the page from which the beacon originated. The u field will always have more information than the url field. 

 

If u is 
http://www.soasta.com/some/page?name=value
 

 

Then url will be 
http://www.soasta.com.
urlThe possessed and stripped URL of the page from which the beacon originated. For example, 
http://www.soasta.com
.
vThe Boomerang version.

 

Table 7: Beacon Params

The following beacon params sub-group appears in some environments. More information about beacon parameters beginning with the nt_

ElementDescription
nt_con_endConnect End. EPOCH. Maps onto an attribute from the browser's NavigationTiming API's window.performance.timing.connectEnd.
nt_con_stConnect Start. EPOCH. Maps onto an attribute from the browser's NavigationTiming API's window.performance.timing.connectStart.
nt_dns_endDNS End. EPOCH. Maps onto an attribute from the browser's NavigationTiming API's window.performance.timing.domainLookupEnd. Refer to the W3 Navigation Timing Overview
nt_dns_stDNS Start. EPOCH. Maps onto an attribute from the browser's NavigationTiming API's window.performance.timing.domainLookupStart.
nt_domcompDOM Complete. EPOCH. Maps onto an attribute from the browser's NavigationTiming API's window.performance.timing.domComplete.
nt_domcontloaded_endDOM Content Loaded End. EPOCH. Maps onto an attribute from the browser's NavigationTiming API's window.performance.timing.domContentLoadedEventEnd.
nt_domcontloaded_stDOM Content Loaded Start. EPOCH. Maps onto an attribute from the browser's NavigationTiming API's window.performance.timing.domContentLoadedEventStart.
nt_domintDOM Interactive. EPOCH. Maps onto an attribute from the browser's NavigationTiming API's window.performance.timing.domInteractive.
nt_domloadingDOM Loading. EPOCH. Maps onto an attribute from the browser's NavigationTiming API's window.performance.timing.domLoading.
nt_fet_stFetch start. EPOCH. Maps onto an attribute from the browser's NavigationTiming API's window.performance.timing.fetchStart.
nt_first_paint[optional] The time when the first paint happened. EPOCH. On Internet Explorer, this is milliseconds since the epoch, while on Chrome this is seconds.microseconds since the epoch. If you detect a decimal point in this number, multiply it by 1000 to compare it to the other timers.
nt_load_endEnd of Load Event. EPOCH. Maps onto an attribute from the browser's NavigationTiming API's window.performance.timing.loadEventStart.
nt_load_stEnd of Load Event. EPOCH. Maps onto an attribute from the browser's NavigationTiming API's window.performance.timing.loadEventStart.
nt_nav_stNavigation start. EPOCH. Maps onto an attribute from the browser's NavigationTiming API's window.performance.timing.navigationStart.
nt_nav_typeNavigation type as a boolean. Maps onto an attribute from the browser's NavigationTiming API.window.performance.navigation.type.
nt_red_cntRedirect count as a boolean. Maps onto an attribute from the browser's NavigationTiming API's window.performance.navigation.redirectCount.
nt_red_endRedirect End. EPOCH. Maps onto an attribute from the browser's NavigationTiming API's window.performance.timing.redirectEnd.
nt_red_stRedirect Start. EPOCH. Maps onto an attribute from the browser's NavigationTiming API's window.performance.timing.redirectStart.
nt_req_stRequest Start. EPOCH. Maps onto an attribute from the browser's NavigationTiming API's window.performance.timing.requestStart.
nt_res_endResponse End. EPOCH. Maps onto an attribute from the browser's NavigationTiming API's window.performance.timing.requestEnd. 
nt_res_stResponse Start. EPOCH. Maps onto an attribute from the browser's NavigationTiming API's window.performance.timing.responsetStart
nt_spdy[optional] Boolean indicating whether page was loaded via SPDY. 1 if page was loaded over SPDY, 0 otherwise.
nt_ssl_st[optional] TMaps onto an attribute from the browser's NavigationTiming API's window.performance.secureConnectionStart.
nt_unload_endEnd of Unload Event (if captured). EPOCH. Maps onto an attribute from the browser's NavigationTiming API's window.performance.timing.unloadEventEnd. 
nt_unload_stStart of Unload Event (if captured). EPOCH. Maps onto an attribute from the browser's NavigationTiming API's window.performance.timingun.loadEventStart. 
The remaining fields in this table represent Resource Timer timestamps. Resource timers are identified by the rt. prefix and apply to an individual resource.
rt.abld[optional] This parameter will be there (but have no value) if the onbeforeunload event fires before the onload event fires. This can happen, for example, if the user left the page before it completed loading.
rt.bmr.{...}
[optional] The rt.bmr.{...} parameters include resource timing information for the boomerang.js request (i.e., how long did the boomerang take to load, etc.)
rt.bmr.conEn
[optional] The connection end timestamp for the boomerang itself.
rt.bmr.conSt[optional] The connection start timestamp for the boomerang itself.
rt.bmr.domEn[optional] The DOM end timestamp for the boomerang itself.
rt.bmr.domSt[optional] The DOM start timestamp for the boomerang itself.
rt.bmr.fetSt[optional] The fetch start timestamp for the boomerang itself.
rt.bmr.resEn[optional] The resource end timestamp for the boomerang itself.
rt.bmr.resSt[optional] The resource start timestamp for the boomerang itself.
rt.bmr.reqSt[optional] The request start timestamp for the boomerang itself.
rt.bmr.secSt[optional] The secureConnectionStart timestamp for the boomerang itself.
rt.bstartThe timestamp when boomerang started executing.
rt.cstart[optional] The start time stored in the cookie if different from rt.tstart.
rt.cnf.{...}
[optional] The rt.cnf.{...} parameters are timestamps for the config.js request.
rt.cnf.conEn[optional] The connection end timestamp for the configuration itself.
rt.cnf.conSt[optional] The connection start timestamp for the configuration itself.
rt.cnf.domEn[optional] The DOM end timestamp for the configuration itself.
rt.cnf.fetSt[optional] The fetch start timestamp for the configuration itself.
rt.cnf.reqSt[optional] The request start timestamp for the configuration itself.
rt.cnf.resEn[optional] The resource end timestamp for the configuration itself.
rt.cnf.resSt[optional] The resource start timestamp for the configuration itself.
rt.cnf.secSt[optional] The secureConnectionStart timestamp for the configuration itself.
  
rt.endThe timestamp when the t_done timer ended (rt.end - rt.tstart === t_done)
rt.oboThe off-by-one can be any number from 0 to rt.sl  The same as oboPages in the session object.
rt.ntvuThis parameter will be there (but have no value) if the onbeforeunload event fires before the page ever became visible. This can happen if the user opened the page in a background tab, and closed it without viewing it, and also if the page was pre-rendered, but never made visible. Use this to check your pre-render success ratio.
rt.quitThis parameter will be there (but have no value) if the beacon was fired as part of the onbeforeunload event. This is typically used to find out how much time the user spent on the page before leaving, and is not guaranteed to fire.
rt.siThe session ID.
rt.slThe Session Length (counter) on pages as an integer.
rt.ssThe Session Start Time. EPOCH
rt.srstThis parameter will be there if the session restarted after being timed out.
rt.startThis specifies where the start time originated. May be one of cookie for the start cookie, navigation for the W3C navigation timing API, csi for older versions of Chrome or gtb for the Google Toolbar. Method used for page 'start' time navigation. Alternately, specify none or cookie
rt.ttThe Total Session Load Time in milliseconds (ms).
rt.tstartThe start time timestamp.

 

Table 8: Session

The following fields are commonly found in the session sub-group.

ElementDescription
IDThe session ID or token string.
latestThe EPOCH latest timestamp for the given beacon. For example, 1404864000314.
pagesThe numner of pages in the session.
oboPagesUnknown.
startThe EPOCH start timestamp for the start of the session. For example, 1404864000314.
totalLoadTimeThe total load time across the session. Same as rt.tt.
isUnLoadThis is a boolean stating whether the beacon was a load beacon or an unload beacon. 1 for unload, 0 for load.

 

Table 9: Custom Metrics

The fields found in the custom_metrics sub-group are all user-created.

ElementDescription
custom_metricsThis parameter is a map from the custom metric number to its value. For example: `custom_metrics:{"0":11,"1":5,"2":1313}

 

Table 10: Warnings

The fields found in the warnings sub-group are those HTTP warnings that occurred in the scope of the beacon session.

ElementDescription
warningsHTTP warnings, such as those that occur during referrer authentication, are posted here. For example, referrer-url-mismatch.

 

Attachments

    Outcomes