Search Performance Evalutation

From Gcube Wiki
Jump to: navigation, search


Introduction

In this section a number of varied queries submitted in the production infrastructure, along with the response times of the gCube system are recorded. The target is to evaluate the performance of the search functionality provided by D4science services.

When submitting a query from the D4science portal, the procedure that is followed until the first results are retrieved is the following:

  • The Application Support Layer constructs the query according to the SearchMaster Syntax
  • It communicates with IS, in order to retrieve the EPRs for the search master running instances
  • The query is submitted to the SearchMaster
  • SearchMaster finds and returns a ResultSet EPR
  • The results are retrieved


Available Resources

During the time the testing was performed, the available resources in the infrastructure were the following:


  • AATSR Products
    • EM Advanced Along-Track Scanning Radiometer (AATSR) Data Set: Daily Products
      • Number of records: 1320
    • EM Advanced Along-Track Scanning Radiometer (AATSR) Data Set: Monthly Products
      • Number of records: 46
    • EM Advanced Along-Track Scanning Radiometer (AATSR) Data Set: Products
      • Number of records: 18.032
    • EM Advanced Along-Track Scanning Radiometer (AATSR) Data Set: Yearly Products
      • Number of records: 5
  • Earth Images
    • EM ESA Earth Images
      • Number of records: 1270
  • Key Documents and Reports
    • EM ESA special issues Living Planet Programme and Earth Observation Handbook
      • Number of records: 4
    • EM Mediterranean Action Plan (MAP) Technical Reports
      • Number of records: 178
  • Meris Level-3 Products
    • EM MERIS Level-3 Chlorophyll in water
      • Number of records: 7
    • EM MERIS Level-3 Global Vegetation Index
      • Number of records: 7
  • Landsat7 Products
    • EM Landsat7
      • Number of records: 12
  • European Environment Agency Reports
    • European Environment Agency: Brochures
      • Number of records: 12
    • European Environment Agency: Corporate Documents
      • Number of records: 38
    • European Environment Agency: EEA Briefings
      • Number of records: 16
    • European Environment Agency: EEA Reports
      • Number of records: 38
    • European Environment Agency: Environmental Assessment Reports
      • Number of records: 12
    • European Environment Agency: Environmental Issue Reports
      • Number of records: 45
    • European Environment Agency: Environmental Monographs
      • Number of records: 5
    • European Environment Agency: Expert's Corners
      • Number of records: 3
    • European Environment Agency: Other Reports
      • Number of records: 90
    • European Environment Agency: State of the Environment Reports
      • Number of records: 7
    • European Environment Agency: Technical Reports
      • Number of records: 100
    • European Environment Agency: Topic Reports
      • Number of records: 100


Query Response Times Sum-up Table

Query Explanation (Physical Language) Query Submitted to SearchMaster (SearchMaster Syntax) Time Until Search Responds Time Until First Result is Retrieved
Search for the word "earth", anywhere, in EM ESA Earth Images collection, in eiDB schema and english language. retrievemetadata as 'eiDB' in 'ENGLISH' on (fulltextsearch by 'earth' in 'ENGLISH' on '94d02c10-f2eb-11dd-a6f4-add312924299') 3 seconds 4 seconds
Search for the word "europe", inside the title, in EM ESA Earth Images collection, in eiDB schema and english language. retrievemetadata as 'eiDB' in 'ENGLISH' on (fieldedsearch by 'title' contains 'europe' in 'ENGLISH' on '778b3e50-f2a6-11dd-8d14-904a5718339a' as 'eiDB' 2 seconds 3 seconds
Search for the word "Atlantic", inside the abstract, in EM Advanced Along - Track Scanning Radiometer (AATSR) Data Set Products collection, in ISO19115 schema and english language. retrievemetadata as 'ISO19115' in 'ENGLISH' on (fieldedsearch by 'abstract' contains 'Atlantic' in 'ENGLISH' on 'f8fea990-00c5-11de-8e55-ae4f02e984c5' as 'ISO19115') 3 seconds 6 seconds
Search for the word "Atlantic", inside the abstract, in EM Advanced Along - Track Scanning Radiometer (AATSR) Data Set Products collection or the word "european", anywhere, in EM Advanced Along - Track Scanning Radiometer (AATSR) Data Set: Monthly Products collection, in ISO19115 schema and english language. Sort the results by title. sort 'ASC' by '//*[local-name()=\'title\']' on (retrievemetadata as 'ISO19115' in 'ENGLISH' on (merge on (fieldedsearch by 'abstract' contains 'Atlantic' in 'ENGLISH' on 'f8fea990-00c5-11de-8e55-ae4f02e984c5' as 'ISO19115') and (fulltextsearch by 'european' in 'ENGLISH' on '0057b970-1ed5-11de-8141-e3edd323bdf8') )) Never (SocketTimeOut Exception)
Search for the word "Atlantic", inside the abstract, in EM Advanced Along - Track Scanning Radiometer (AATSR) Data Set Products collection or the word "european", anywhere, in EM Advanced Along - Track Scanning Radiometer (AATSR) Data Set: Monthly Products collection, in ISO19115 schema and english language. retrievemetadata as 'ISO19115' in 'ENGLISH' on (merge on (fieldedsearch by 'abstract' contains 'Atlantic' in 'ENGLISH' on 'f8fea990-00c5-11de-8e55-ae4f02e984c5' as 'ISO19115') and (fulltextsearch by 'european' in 'ENGLISH' on '0057b970-1ed5-11de-8141-e3edd323bdf8') ) 11 seconds 14 seconds
Search for the word "initiative", inside the abstract, in all the collections of AATSR Products group, in ISO19115 schema and english language. merge on (retrievemetadata as 'ISO19115' in 'ENGLISH' on (fieldedsearch by 'abstract' contains 'initiative' in 'ENGLISH' on 'f8fea990-00c5-11de-8e55-ae4f02e984c5' as 'ISO19115')) and (retrievemetadata as 'ISO19115' in 'ENGLISH' on (fieldedsearch by 'abstract' contains 'initiative' in 'ENGLISH' on 'f519f550-00c5-11de-8e55-ae4f02e984c5' as 'ISO19115')) and (retrievemetadata as 'ISO19115' in 'ENGLISH' on (fieldedsearch by 'abstract' contains 'initiative' in 'ENGLISH' on '0bfd7c60-00c6-11de-8e55-ae4f02e984c5' as 'ISO19115')) and (retrievemetadata as 'ISO19115' in 'ENGLISH' on (fieldedsearch by 'abstract' contains 'initiative' in 'ENGLISH' on '1030ece0-00c6-11de-8e55-ae4f02e984c5' as 'ISO19115')) 7 seconds 8 seconds
Search for the word "satellite", inside the title, or the word "europe", inside the description, in EM ESA Earth Images collection, in es schema and english language. Sort the results by title. sort 'ASC' by '//*[local-name()=\'title\']' on (retrievemetadata as 'es' in 'ENGLISH' on (merge on (fieldedsearch by 'title' contains 'satellite' in 'ENGLISH' on '778b3e50-f2a6-11dd-8d14-904a5718339a' as 'es') and (fieldedsearch by 'description' contains 'europe' in 'ENGLISH' on '778b3e50-f2a6-11dd-8d14-904a5718339a' as 'es') )) 33 seconds 35 seconds
Search for the word "satellite", inside the title, or the word "europe", inside the description, in EM ESA Earth Images collection, in es schema and english language. retrievemetadata as 'es' in 'ENGLISH' on (merge on (fieldedsearch by 'title' contains 'satellite' in 'ENGLISH' on '778b3e50-f2a6-11dd-8d14-904a5718339a' as 'es') and (fieldedsearch by 'description' contains 'europe' in 'ENGLISH' on '778b3e50-f2a6-11dd-8d14-904a5718339a' as 'es') ) 5 seconds 26 seconds
Search for the word "Antarctica", inside the description, and the word "map", inside the description, in EM ESA Earth Images collection, in es schema and english language. retrievemetadata as 'es' in 'ENGLISH' on (join inner by 'DocID' on (fieldedsearch by 'description' contains 'Antarctica' in 'ENGLISH' on '778b3e50-f2a6-11dd-8d14-904a5718339a' as 'es') and (fieldedsearch by 'description' contains 'map' in 'ENGLISH' on '778b3e50-f2a6-11dd-8d14-904a5718339a' as 'es') ) 5 seconds 28 seconds
Search for the word "satellite", anywhere, inside any collection. fulltextsearch by 'satellite' in on 9 seconds 11 seconds
Search for the word "Accumulation", inside the title, and the word "transformation", inside the title, in EM Mediterranean Action Plan (MAP) Technical Reports collection, in es schema and english language. retrievemetadata as 'es' in 'ENGLISH' on (join inner by 'DocID' on (fieldedsearch by 'title' contains 'Accumulation' in 'ENGLISH' on '1ca21eb0-076d-11de-be1f-cda49fe94041' as 'es') and (fieldedsearch by 'title' contains 'transformation' in 'ENGLISH' on '1ca21eb0-076d-11de-be1f-cda49fe94041' as 'es') ) 4 seconds 6 seconds
Search for the word "Environment", inside the title, in EM Mediterranean Action Plan (MAP) Technical Reports collection, in dc schema and english language. Sort the results by publisher. sort 'ASC' by '//*[local-name()=\'publisher\']' on (retrievemetadata as 'dc' in 'ENGLISH' on (fieldedsearch by 'title' contains 'Environment' in 'ENGLISH' on '1ca21eb0-076d-11de-be1f-cda49fe94041' as 'dc')) 3 seconds 5 seconds
Search for the expression "e*", inside the title, in all the collections of EM Demo vre, in es schema and english language. merge on (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on '1ca21eb0-076d-11de-be1f-cda49fe94041' as 'es')) and (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on 'f519f550-00c5-11de-8e55-ae4f02e984c5' as 'es')) and (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on '0bfd7c60-00c6-11de-8e55-ae4f02e984c5' as 'es')) and (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on 'f8fea990-00c5-11de-8e55-ae4f02e984c5' as 'es')) and (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on '1030ece0-00c6-11de-8e55-ae4f02e984c5' as 'es')) and (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on '778b3e50-f2a6-11dd-8d14-904a5718339a' as 'es')) and (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on 'adab1920-f2a5-11dd-8d14-904a5718339a' as 'es')) and (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on 'd793c130-f91f-11dd-8e3c-ae4f02e984c5' as 'es')) and (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on 'e90a9f00-f920-11dd-8e3c-ae4f02e984c5' as 'es')) and (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on '62c2a530-f2ec-11dd-8d16-904a5718339a' as 'es')) and (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on '24ce2920-09a8-11de-8d6f-a19cf0b0d020' as 'es')) and (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on 'a1c36100-09aa-11de-8d70-a19cf0b0d020' as 'es')) and (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on 'd0e1b990-076c-11de-be1f-cda49fe94041' as 'es')) and (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on 'b1d74810-0769-11de-be1f-cda49fe94041' as 'es')) and (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on '4c619c10-09cb-11de-8f05-a19cf0b0d020' as 'es')) and (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on '6884de10-09cc-11de-8f05-a19cf0b0d020' as 'es')) and (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on 'dd7f83f0-09d1-11de-8f06-a19cf0b0d020' as 'es')) and (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on 'b7430090-09d1-11de-8f06-a19cf0b0d020' as 'es')) and (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on '7a8f62a0-09c8-11de-8f05-a19cf0b0d020' as 'es')) and (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on '2ee43fc0-09b3-11de-8d7d-a19cf0b0d020' as 'es')) and (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on '3a872600-09c0-11de-8f02-a19cf0b0d020' as 'es')) and (retrievemetadata as 'es' in 'ENGLISH' on (fieldedsearch by 'title' contains 'e*' in 'ENGLISH' on 'a52c31e0-09d3-11de-8f0c-a19cf0b0d020' as 'es')) 45 seconds 50 seconds
Search for the word "greece", or the word "envisat", anywhere, in every collection. fulltextsearch by 'greece' , 'envisat' in on 9 seconds 10 seconds

Query Response Times Sum-up Table v2 - SearchMaster readings (rel1.7)

Testing environment

VRE
/EM/GCM
Search GHN
node19.p.d4science.research-infrastructures.eu
log4j default level
INFO
RS EPR Cache
disabled
Readings procedure
Median of three readings, taken in a hot search status (not currently performing any reharvest or garbage collection)
Readings
Time until a result set endpoint reference is returned as response. Note that this does not mean that the actual data are generated.


Readings

Query No Planning Time in millis Execution Time in millis Total Time in millis
1 1139
  • 369 Total
    • 65 FTIndex
    • 292 MMFactory
 1525
2 881
  • 763 Total
    • 203 FTIndex
    • 333 MMFactory
 1660
3 1225
  • 1668 Total
    • 82 FTIndex
    • 566 MMFactory
 2909
6 3083
  • 2594 Total (8748 if sequential)
    • 1423 + 791 + 762 + 1095 FTIndex in parallel
    • 149 + 176 + 151 + 156 MMFactory in parallel
    • 958 Merge
 5692
7 1728
  • 16223 Total
    • (1428+2293) FTIndex in parallel
    • 5084 Merge
    • 133 MMFactory
    • 8660 Sort
 17966
7 (if terms are in *) 1661
  • 3527 Total
    • (93+58) XMLIndexer in parallel
    • 236 Merge
    • 3154 Sort
 5205
8 1613
  • 2419 Total
    • (344+1024) FTIndex in parallel
    • 1192 Merge
    • 174 MMFactory
 4049
9 1683
  • 534 Total
    • (143+103) FTIndex in parallel
    • 278 Join
    • 106 MMFactory
 2233
10 2066
  • 11754 Total
    • (1522+613+585) FTIndex in parallel
    • 10192 DIR-Merge
 13837
13 1121
  • 34128 Total (almost the same if sequential!)
    • ~30000 FTIndex (resourcex18)
    • ~700 MMFactory (RIx18)
    • 3797 Merge
 44920
13 (if *e* instead of e*) 9682
  • 5730 Total (21801 if sequential)
    • ~600 XMLIndexer (resourcex18)
    • 4408 Merge
 15429

Comments

Specific Services

XMLIndexer
Faster than FTIndex in most cases. Very good in low load. Problem in multiple, parallel invocations in the same node. At heavy loads, it gets stuck for some minutes until some cleaning procedure takes place. E.g. XMLIndexer service stuck after 6 parallel invocations. MMFactory, running at the node node, was also stuck.
FTIndex
In some nodes it's extremely slow (more than one second). In other nodes it is faster but again not as fast as it should be. This means that there is CPU or memory overload. Further analysis is necessary. FTIndex resources can cope with huge loads but with large delays (@query no13, each FTIndex resource responded after 30sec).
DIR
Not as fast as it should be. It may not operate in non-blocking mode so its delay may be the result of waiting for the full input RS payload to become available.
Merge
Shows significant delay at producing the first result set part.

General

  • Irregularities in performance, especially in large queries (such as No 13). We got back execution times that range from 17 seconds up to INF (resources didn't respond at all).
  • Irregular response time of RS. Significant delay of reading the first RS part.
  • We still need to stress-test IndexFuse operator.
  • Rarely IS seem to be stuck (completely unresponsive). IS Monitoring did not show any RIs or resources. IS comes back after some minutes (usually 15').
  • Parallelization of MMFactory (merge of many MMFactory invocations instead of a single invocation) leads to great performance enhancement.
  • In the current scheme, FTIndexes are more or less useless, since XMLIndexer resources deliver data much faster than the FTIndexer (let alone the combination of MMFactory and FTIndexer).

Search Framework

See here

Retrieved from "https://wiki.gcube-system.org/index.php?title=Search_Performance_Evalutation&oldid=25252"