Per the xAPI specification, data retrieval via from an LRS is paged to 500 statements at a time for one query. While you can use the 'more' link approach, retrieving even a few hundred thousand statements will take a while if you do it one batch at a time in order. If you need to do it frequently, there are a few options:
- Parallelize retrieval. If you know, for example, a list of agent identities or activities in the data, you can get data from each of those identities in parallel in 500 statement chunks. Another way, if you know the approximate time range of the data, is to start multiple retrieval processes using since and until parameters on the statements API to slice up the time range, again retrieving in parallel.
- Periodic bulk data retrieval. Instead of parallel retrieval, we're comfortable with very occasionally providing a complete archive of your statements; it is your data, after all. That isn't an API call, but something we'd be happy to do for you up to a few times a year. Contact us.
- Syncing new statements. Whether you parallelize retrieval or not, once you've got the statements, from then on you only need to retrieve what's necessary to sync any new statements. If you retrieve statements by stored periodically, and always provide the most recent 'x-experience-api-consistent-through' header you've received from a retrieval as the 'since' parameter, you'll be guaranteed you won't miss any statements, and will only get statements since you last retrieved plus a buffer. To avoid storing duplicate statements, just make sure you only have one of each statement ID.
- Wax LRS warehouse views. If you regularly need to retrieve or process statements in very large quantities but maintaining a sync process then parsing on your side isn't a good option, or if you'd just like to simplify things and have more flexibility, on our Impact plan we offer SQL Data Warehouse access to all your data - you can connect to a dedicated data warehouse using a business intelligence tool (such as Tableau), ETL tool, or custom code using JDBC, ODBC, or a variety of database drivers. The database has your data arranged into easy to use tables that make slicing and dicing through your data using analytical queries over huge numbers of statements fast. It is also suitable for regular bulk retrieval by any custom conditions you want, since you can use any SQL you want. The data will also come back in the form you want -- you can retrieve the raw statements, or you can compute a perfect set of fields for a specific purpose. Contact us for a demo and discussion.
Value to Customer Learning Ecosystem: Connect activity reporting about learner experience from Wax LRS to a BI tool or custom reporting platform. Make it easier for other systems to consume LRS data for further usage and analysis.
Value to Consulting Partner Learning Services: Connect Wax LRS as a data source to your BI tools for further analysis and reporting.
Additional data access options: We also offer Wax LRS specific APIs for accessing Wax report data or you can generate and download Historical Rollup Reports in csv format from the Wax LRS dashboard for any time period. We know and believe it's important to provide you a variety of options for accessing your data, let us know if you want to chat more!
Related note: If you are in the European Union (EU) and intend to store personal information about EU citizens using Wax LRS, you need to comply with EU data protection laws (EU Data Protection Directive 95/46/EC). We can help! Saltbox has prepared an EU Model Clauses agreement for existing and new Wax LRS customers to facilitate the transfer of personal data outside of the EU in accordance with the EU data privacy laws. Contact us to learn more.