Shelf Label Server

Introduction#

The Electronic Shelf Label (ESL) server is a component of Novos that provides near real-time ACE pricing data suitable for ingestion by an ESL system.

System Overview#

SmartShop Flow

Primary System Components#

4690 Store Controller#

The source of information for the ESL server. The store controller stores the ACE POS data files that drive ACE pricing and are used as input to the ESL server process.

Novos Server#

Novos Server monitors the data files on the controller and when a change is detected the file is downloaded, processed and stored in Novos's local database on disk.

Processing of the data files includes individual item change detection (only changed or new items are processed) as well as aggregation of information from the multiple data files into a single ESL record for each item in the ACE pricing database.

ESL Clients#

The ESL server supports both event driven clients (clients which prefer to be notified asynchronously when an ESL changes) and polling based clients (clients which prefer to receive ESL change data in response to a client request). For both client types a mechanism is provided that allows only changed records to be exchanged between the server and the client.

For event based clients, each event contains a list of ESL records that have changed since the previous change event was generated.

For polling based clients, the client may specify a timestamp on the request for data. Only records changed after that point in time will be transmitted.

Note: The timestamp on the polling interface is optional. If no timestamp is provided by the client the server will transmit all ESL records. This is useful when the client wishes to do an initial load of the data or when the client determines that it is out of sync with the server and wants to start with a full download.

Operation#

The ESL functionality is designed to run alongside the QVS Smart Shopping API functionality. As such a single instance of Novos server running against a store controller can serve both SSAPI clients such as smart shopping carts as well as ESL clients.

This section describes in more detail how the ESL functionality works.

Data File Change Detection#

The first phase of generating ESL data involves monitoring the data files on the controller for changes. In order to minimize the impact this operation has on the controller Novos uses a combination of file system checks and data checks to determine if a data file has changed.

For keyed files such as the item record file, Novos monitors the keyed file performance statistics of the file and pulls the file locally whenever a change in the number of record adds or record modifications is noticed. For non-keyed file the file system size and timestamp are monitored and the file is pulled whenever either changes.

Novos uses 4690 TCC by default to download the file once a change is detected. It can also use SFTP to transfer the file (see Configuration section). SFTP is considerably faster than TCC for large files. Initial testing in the QVS lab indicates SFTP can transfer the item record file in 1/4 the time it takes using TCC. As file transfer time is like the single largest component in overall time elapsed from the time data is changed until the ESL listener is notified, switching to SFTP transfer can significantly improve response times.

Overall response time will be a combination of the polling frequency, the file transfer method (TCC or SFTP) as well the time to analyze/diff the data (see Data Diff and Local Storage section below). With the default 60 second polling interval coupled with TCC transfer time and approximately 300,000 items, changes were detected and clients were notified approximately 5 minutes after the change was performed.

This time is for illustrative purposes only and performance will vary based on the specific hardware used. The most optimistic estimate with a short polling interval (10 seconds) and SFTP file transfer is probably on the order of approximately 2 minutes between data change and client notification.

Data Diff and Local Storage#

The file monitor process is only checking for the existence of changed data, it does not detect which specific records have changed until the data is pulled from the controller to Novos cache. This behavior is designed so that the there is no burden on the controller other than that required to download the file.

The potentially time-consuming and resource intensive task of determining what records changed is done offline from the controller.

The process involves going through each item in the cached data file and comparing it to the previously cached version and looking to see if any data fields changed. Once a change is detected the data is stored in a local SQL database.

Realtime notifications as well as polling requests from clients are served from the SQL database.

Realtime Listener Updates#

For each batch of changes detected Novos sends an HTTP PUT event to each subscribed client's endpoint. The client subscribes to these events by sending Novos an HTTP PUT with a JSON object that indicates what URL to post when changed data is detected. Other optional fields can be set in the JSON object to control how data is returned to the client. See "Running the Sample Client" for an example.

The data sent is a JSON array of ESL records that have changed.

Realtime changes are not queued for clients that are not online at the time the changes are detected. If a client goes offline or shuts down after subscribing to realtime updates the client not Novos is responsible for making sure it's database is synchronized properly. Each client should use the polling interface each time it initializes or comes online to ensure it has the most current data.

Novos will attempt to notify each listener of each update once. If Novos notices it getting persistent errors posting updates to a client it will remove that client from the list of subscribers. Novos keeps an indicator for each ESL client and stores the time of the last successful update sent to that client. If 4 or more days elapse since the last successful update and the latest attempted update, the client subscription is removed and will have to re-subscribe. A client can also remove itself from the subscribers list by posting JSON to http://<NOVOS_ESL_HOST>:8080/acews/pricedb/ESL/unsubscribe in a manner similar to the method used to create the subscription.

View List Of ESL Realtime Listeners#

A list of the currently subscribed endpoints can be retrieved by browsing to http://<NOVOS_ESL_HOST>:8080/esl-subscribers.html or by issuing a GET to http://<NOVOS_ESL_HOST>:8080/acews/pricedb/ESL/subscribers.

Polling Listener Queries#

Novos also provides a mechanism whereby a client can poll the server for ESL data. The interface is timestamp based. If no timestamp is sent by the client the entire ESL database is sent. If a timestamp is specified only records that have changed after that timestamp are sent. The timestamp format is standard Linux epoch timestamp.

Note: Store number is included with the ESL data sent to the client. Novos must be online to the 4690 controller in order for store number to be known. If Novos is not online to the controller at the time the request is received Novos will return HTTP status 503 Service Unavailable to the client.

Example: Query for all records

curl -X GET http://<NOVOS_ESL_HOST>:8080/acews/pricedb/ESL

Example: Query for records changed after a specific timestamp

curl -X GET http://<NOVOS_ESL_HOST>:8080/acews/pricedb/ESL?timestamp=1632853773299

Realtime ESL Queries#

During testing and for audit purposes Novos provides a browser based interface that allows the user to display an ESL record in realtime outside the normal file monitoring process. To use this interface browse to http://<NOVOS_ESL_HOST>:8080/esl-lookup.html.

You will be presented with a webpage that allows you to enter a UPC and see the ESL for that item. The data returned will be the data as it exists on the controller at that moment. If you change the data on the controller and submit the form again you'll see the changed data.

Note: Due to limitations concurrent access to the local SQL database, realtime queries may fail if the file monitor/change detection process is accessing the database at the same time. If an error occurs you need to wait for the file monitor process to finish before realtime queries will function correctly.

Configuration#

The behavior of the file monitoring process is configured using the properties file itemFileToPriceDB.properties. Usage of this file allows for modifying parameters such as polling frequency, the location and names of which files are monitored and how the files are transferred.

Each setting has a default value and need not be set in the configuration file unless you wish to modify the default value. Configurations changes are not dynamic. Novos will need to be restarted for configuration changes to take effect.

Settings that control file names and file location#

The names and locations of which files to pull are configured seperately.

Setting Name	Description	Default Value
itemFilePath	Path to the location of the item record file to monitor.	R::/ADX_IDT1
itemFileName	8.3 file name of the item record file to monitor. The ESL server expects the format of this file to be the standard ACE 169-byte Item Record file with 14-digit item code.	EAMITEMR.DAT
auxItemFilePath	Path to the location of the optional auxiliary item record file to monitor.	R::/ADX_UDT1
auxItemFileName	8.3 file name of the optional auxiliary item record file to monitor.	Configurable
optionsFilePath	Path to the location of ACE terminal options file.	R::/ADX_IPGM
optionsFileName	8.3 file name of the ACE terminal options file.	EAMOPTNS.DAT
hppFilePath	Path to the location of ACE terminal options definitions file.	R::/ADX_IPGM
hppFileName	8.3 file name of the ACE terminal options definition file.	INCOPTNS.HPP

Settings that control polling behavior#

The file monitor polling behavior can be customized using the following settings.

Setting Name	Description	Default Value
pollingTimeout	How frequently should the monitor check for file changes in milliseconds	60000 (60 seconds)
delayAfterDetect	The poller waits for a quiet period after a change is detected before pulling the file. This is for situations where a batch of changes are made to a file over a short period of time. For example, a user is in Item Data Maintenance changing several records. `delayAfterDetect` is the delay in milliseconds to wait after a change is detected before pulling a file	5000
readThrottle	In order to minimize the burden of downloading the files from the controller, a throttling delay between file reads is introduced. This delay has minimal impact on the transfer time of the file. Typically you would not change this setting unless instructed to by QVS. The delay is specified in milliseconds	10
conflictMaxRetries	Since the files the ESL server is reading on the controller are actively in use by ACE and other processes on the controller, access conflicts can arise when monitoring and downloading the files. An example would be if ACE is updating an item record and has locked the record in exlusive mode at the instant the file is being read and downloaded by the ESL server. If such a conflict arises, the ESL server will retry the download after a short delay to attempt to recover from the conflict. If the server is not able to progress after `conflictMaxRetries` number of retries, the download is aborted and will not be re-attempted until the next polling interval	5

Settings that control large file transfer protocol#

Substantial improvements in file transfer times can be achieved using SFTP rather than 4690 TCC (Terminal-Controller Communications) for large files.

Setting Name	Description	Default Value
fileTransferViaSftp	Setting to true causes the ESL server to download/transfer large files using SFTP protocol	false
username	The SFTP username to use when logging into the SFTP server to initiate file transfer	none
password	The SFTP password to use when logging into the SFTP server to initiate file transfer	none

Settings that control client timeout and purging#

Setting Name	Description	Default Value
waitForClientTimeoutSeconds	server will timeout after this amount of time waiting for client response after update request (in seconds). This timeout should be longer than the pollingTimeout	60
purgeInactiveClientDuration	inactive clients will be purged from update list after this amount of time (d=days, h=hours, m=minutes)	7d

Settings that control TLS certificate validation#

When Novos/ESL service makes an HTTP call to an ESL or PriceDB subscriber using TLS (the subscriber URL uses the HTTPS scheme) the default is for Novos to validate the subscriber's certificate with the host platform's trust store.

Sample Event Based Client#

A sample event based client is provided as a means to exercise the system and become familiar with the operation of the PUB/SUB event based mode of operation. The sample client fulfills the role of the external ESL system in that it provides an HTTP endpoint that the ESL server can post events to.

The sample client is a Java 1.8 based Spring Boot application packaged as a JAR with all dependendencies. You will need to have a Java 1.8 or greater JRE installed in order to run the sample.

Running The Sample Client#

Note: for the purposes of this demonstration, it's assumed that Novos/ESL server is already configured and running. It could be on the same host as the ESL sample client or on a different host. This document will refer to the host running Novos as <NOVOS_ESL_HOST> and the host running the ESL sample client as <ESL_SAMPLE_HOST>

Start the sample client by running the command

java -jar esl-sample-client-1.0.0.jar

After the sample has initialized you can verify that it is running and ready to receive requests from the ESL server by browsing to http:<ESL_SAMPLE_HOST>:8081. You should see the message 'Greetings from ESL sample client!'.

Once you have the sample client running you need to send a message to the Novos/ESL server that subscribes the client to ESL data change events. For this example we'll use curl to send the subscription requests. First create a data file that contains the URL of the client program and name it subdata.json. The file contains a JSON object with one attribute url that contains the endpoint Novos?ESL should post when ESL data is changed or generated.

Sample subdata.json (replace <ESL_SAMPLE_HOST> with the IP address of the host running the sample client):

{   "url":"http://<ESL_SAMPLE_HOST>:8081/ESL"}

Then send that data to Novos/ESL using curl (replace <NOVOS_ESL_HOST> with the IP address of the host running the Novos/ESL server):

curl -H "Content-Type:application/json" -X PUT ` \ <br>  `http://<NOVOS_ESL_HOST>:8080/acews/pricedb/ESL/subscribe -d @subdata.json

At this point the Novos/ESL server is monitoring the 4690 ACE POS files for changes and there is a client subscribed to the ESL endpoint ready to receive ESL data update messages, all that remains is to trigger an update. To do that, log on to the 4960 controller and using ACE item data maintenance, modify an item.

Depending on how the Novos/ESL system is configured (pollingTimeout, SFTP vs. TCC) it could take several minutes. Eventually you should see output from the sample program with the changed ESL information similar to the following:

{  "status": "OK",  "timestamp": 1635246807417,  "action" : "U",  "upcNumber" : "30088117447",  "itemNumber" : "2020202020",  "itemDescription" : "GAVISCON EXT STREN",  "itemSalesTaxFlag" : "",  "customerNumber" : "342",  "salesPriceQuantity" : 1,  "salesPriceAmount" : 909,  "itemUnitOfMeasure" : 909,  "itemUnitOfMeasureDescription" : "100 CNT",  "itemPackQuantity" : 4,  "itemSizeDescription" : "100 CT",  "loyaltyNetSalesPriceAmount" : 0,  "loyaltyLimitQuantity" : 0,  "loyaltyDiscountAmount" : 0,  "loyaltyUnitOfMeasure" : 0,  "loyaltyRequiredQuantity" : 0,  "loyaltyNetSalesPriceQuantity" : 0,  "primarySavingsAmount" : 60  "restrictedSalesType" : 0,  "couponItemCode" : "",  "flags" : "N...................",  "userExit2ItemCode" : "32021",  "userExit2ItemDescription" : "PHLBEVTAX RECOVERY",  "userExit2Price" : 21}Received 1 ESL updates

Optional fields in the subscription request allow for some control over how data is returned to the client. Here is another sample subdata.json with all the optional fields set:

{   "url": "http://<ESL_SAMPLE_HOST>:8081/ESL",   "max-message-length": 100000,   "headers": [      {         "name": "header1",         "value": "value1"      },      {         "name": "header2",         "value": "value2"      }   ],   "template": {"key":null,"value":{"type":"JSON","data":[]}}}

The max-message-length field specifies the max message size the client can accept in bytes.

The array of headers allow clients to define custom headers to be used in the return URL. For example:

{   "name": "Authorization",   "value": "Bearer <token>"}

Clients can also specify how they want the return data to be formatted in JSON by using the template field. This field must be a valid JSON object with a valid array field named "data". All data records will be put in this array.

ESL Record Fields#

(All monetary values represent the amount in cents)

status#

Overall status of this record

OK
UNSUPPORTED_PRICING_METHOD
UNSUPPORTED_COUPON_TYPE
EXCEPTION

timestamp#

The Linux/Unix timestamp value (seconds since the epoch) marking the time when this item was last modified in the price database

action#

Last action applied to this record in the database.

I: Item was inserted
U: Item was updated
D: Item was deleted

upcNumber#

The item's UPC number from the item record file

itemNumber#

Item identifier

itemDescription#

Item description from the item record file

itemSalesTaxFlag#

Flag indicating which tax plans apply to this item or blank if no tax plans apply ACE uses taxs plans A-H so a value of "AB" would indicate that two tax plans A and B apply to this item

customerNumber#

Store number from the store controller

salesPriceQuantity#

Sales quantity that applies to salesPriceAmount from the item record file

salesPriceAmount#

Unit price from the item record file

itemUnitOfMeasure#

The per unit of measure price for the item

itemUnitOfMeasureDescription#

Description of the items unit of measure

itemPackQuantity#

Package quantity of the item

itemSizeDescription#

Description of the package size

loyaltyNetSalesPriceAmount#

Price of the item including the loyalty coupon savings amount

loyaltyLimitQuantity#

Maximum quantity eligible for loyalty discount savings

loyaltyDiscountAmount#

Amount of Price Plus savings

loyaltyUnitOfMeasure#

The per unit of measure price for the item with the loyalty discount

loyaltyRequiredQuantity#

The must-buy quantity in order to qualify for the loyalty discount

loyaltyNetSalesPriceQuantity#

Sales quantity that applies to loyaltyNetSalesPriceAmount

primarySavingsAmount#

The difference between SalesPriceAmount and the comparison price in the item record file

restrictedSalesType#

The restricted sale type code for the item or 0 if item is not a restricted sale item.

couponItemCode#

The item code of the coupon attached to this item.

flags#

Flags convey additional properties about this item

Offset 0
Y=requires loyalty card to trigger.
N=coupon triggers without loyalty card
Offset 1
Y=item sold by weigth.
N=item not sold by weight
Offset 2-19
Reserved

userExit2ItemCode#

The value of the integer store in the USREXIT2 field of the ACE item record file as documented in the Toshiba ACE Programming Guide.

userExit2ItemDescription#

If there exists an item in the Ace item record file whose item coded matches the value of the ESL record's userExit2ItemCode value then the description of that item is stored userExit2ItemDescription. Otherwise, empty string.

userExit2Price#

If there exists an item in the Ace item record file whose item coded matches the value of the ESL record's userExit2ItemCode value then the unit price of that item is stored userExit2Price. Otherwise, zero.

departmentNumber#

Department number from the item record file for this item