Difference between revisions of "Accounting Aggregator"

Revision as of 12:01, 2 January 2018

This component is responsible of aggregating the collected Accounting data.

Accounting Aggregator in the Accounting Architecture

The following image evidences the accounting-aggregator components as part of Accounting Architecture:

Accounting Aggregator Execution

This component is a Smart Executor plugin and is launched as Global Scheduled Task (see Smart Executor for further information)

It aggregates (lossless) accounted Usage Record. The scheduling plan is to run on different smart-executor equipped with this plugin with the following parameters:

• Hourly => accounting records per hour are available for the past week;
• Daily => accounting records per day are available for the past 3 months;
• Monthly => accounting records per month are available for the past 3 years;
• Yearly => accounting records per year are available for the past 10 years;

How to manage calledMethod lead to ServiceUsageRecord cannot be aggregated

RESTFull (and REST-like) services contain the ID of the managed resource inside the path of the URL. Smartgears auto calculate calledMethod by using the relative path of the URL. This lead to have ServiceUsageRecord cannot be aggregated.

To solve such an issue some actions are need from developers:

• Rewrite calledMethod in the service code
• Provide regular expression to match old accounted records to allow to rewrite them from accoutnign-aggrator

=== Rewrite calledMethod ===calledMethod

The developer have to insert as first line of the entry method the following line of code to instruct Smartgears to set the desired calledMethod

CalledMethodProvider.instance.set("rewrittenCalledMethod");

Resource Registry is an example of how to properly use such a feature for RESTFull services.

For example Resource Registry use the following URLs to read/update/delete a Facet having UUID 4023d5b2-8601-47a5-83ef-49ffcbfc7d86

GET /resource-registry/er/facet/4023d5b2-8601-47a5-83ef-49ffcbfc7d86

PUT /resource-registry/er/facet/4023d5b2-8601-47a5-83ef-49ffcbfc7d86

DELETE /resource-registry/er/facet/4023d5b2-8601-47a5-83ef-49ffcbfc7d86

If the service does not rewrite the calledMethod we have two effect:

• the three operation cannot be distinguished from accounting and lead to aggregate the records operating to the same facet.
• the uuid does not allow to aggregate the operation across facets.

Resource Registry instruct smartgears to rewrite the calledMethod as following:

CalledMethodProvider.instance.set("GET /resource-registry/er/facet/ID");

CalledMethodProvider.instance.set("PUT /resource-registry/er/facet/ID");

CalledMethodProvider.instance.set("DELETE /resource-registry/er/facet/ID");

I such a way the different methods are distinguishable and the ServiceUsageRecords can be aggregated across facets.

Provide regular expression

Accounting provide a way to rewrite the the old calledMethod to be compliant with the one declared in the service by using CalledMethodProvider.

The developer has to provide the following a json declaring the service class and name to match to check if the provided regular expression match the old calledMethod so that is rewritten with the provided replacement.

For example to properly manage the following ServiceUsageRecord

{
	"operationCount": 1,
	"creationTime": 1507334430724,
	"serviceClass": "Index",
	"callerHost": "88.197.53.47",
	"callerQualifier": "UNKNOWN",
	"recordType": "ServiceUsageRecord",
	"consumerId": "UNKNOWN",
	"aggregated": true,
	"serviceName": "FullTextIndexNode",
	"duration": 3,
	"maxInvocationTime": 58,
	"scope": "/d4science.research-infrastructures.eu/gCubeApps/iSearch",
	"host": "dewn10.madgik.di.uoa.gr:8080",
	"startTime": 1507334430481,
	"id": "12a631da-1541-4771-8961-8d7e4a694bc0",
	"calledMethod": "/delete/3121eccdcbf9c6e982dcc5a2164b8b4029f51d8b1c448ddaf46316178d755c6f/oai:oai.datacite.org:352019",
	"endTime": 1507334430481,
	"minInvocationTime": 1,
	"operationResult": "SUCCESS"
}

an example of json to provide is

{
	serviceClass : "Index"
	serviceName : "FullTextIndexNode"
	regex : "/delete/[^\\t\\n\\r\\f\\v]+/[^\\t\\n\\r\\f\\v]+"
	replace : "/delete/COLLECTION_ID/ITEM_ID"
}

Please note that RESTFull services like the Resource Registry cannot properly managed in this way so it is important that the developer fixes the service as soon as possible. This is due to missing operation type in calledMethod which is not part of the URL (as in FullTextIndexNode) instead was provided from HTTP method.