root/doc/RATING.txt

CDRTool rating engine
---------------------

CDRTool provides on-the-fly rating of CDRs from multiple data-sources like
OpenSIPS, Asterisk or Cisco gateways based on easy to build rating plan,
which can be provisioned in a web interface or imported from csv files.

A Call Detail Record (CDR) is one record from the radius radacct table. The
CDR contains all information related to a session, its duration, the calling
and called party and media information. The rating engine calculates the
price of the session. The calculation is done once and its results are saved
in the CDR table for later use. It is possible to re-calculated the prices
at a later time for example when having to change the tariffs.

Based on exceptions, different rates may be applied per caller party
identified by source IP, domain or subscriber.

Rating is applied only after the call has ended. CDRTool considers that a
call has ended when there is a stop time. 

In case of missing BYEs, CDRTool relies on the fact that MediaProxy will
update the CDR with the proper stop time information based on the last
moment the media stream passed through the media proxy. This functionality
is not available when the end-points have negotiated using ICE another
candidate than MediaProxy.

The rates are linked with profiles corresponding with different time of the
day, day of the week or holidays. For rating calls, which span multiple
profiles, the right rate is selected and applied for the call duration
within each profile. Each customer may be assigned its own dedicated rating
plans destination id and names. Chained profiles are possible to enable
exception based rating. Multiple customers may share a common rate list,
while some destinations may be rated differently, only the differences must
be provisioned. Multiple time zones are supported for multiple billing
parties hosted on the same platform.


Rating logic
------------

Different customers can have different rating plans. A rating plan is a
unique combination of holidays, day of week, time of day, destination ids,
and associated costs.

The following steps are performed to rate a CDR:

1. Determination of the billing party
2. Determination of the destination id
3. Determination of the costs

The steps are described in detail below.


1. Determination of the billing party
-------------------------------------


To be able to calculate the Price for a call the rating engine must
determine whose rating plan to use.

The rating engine does this by performing a match against entries in the
billing_customers table for the the BillingPartyId field of the the CDR
(radacct.UserName for radius based datasources) in the following order:

a. SIP account user@domain 
b. SIP domain of the SIP account
c. Source IP of the session
d. Default (when none of the above matches)

The first match is considered to be the billing party for which the rating
plan is determined. 

The rating plan is further derived from the profiles associated with the
entry found in the billing_customers.

Note

The billing_customers table field that matches the Source IP is called
gateway. "gateway" or "trusted peer" terms are used interchangeably in this
document. They both relate to the source IP address that generated the SIP
session.

If you want to use the multi-tenant features for OpenSIPS accounting you
must create an integer reseller_id column in the trusted and domain tables.
Traffic generated by IP addresses from the trusted peers table and local
domains will be marked as belonging to their correspondent reseller_id. The
reseller_id is then later used to lookup destinations and rates belonging to
the reseller_id.


2. Determination of the destination id
--------------------------------------

The rating engine identifies the 'destination id', which has associated
rates depending on day of week and time of day.

The 'destination id' is for example a country prefix like '31' for the
Netherlands. This 'destination id' has prices associated with it so all
calls to the Netherlands will have prices associated with prefix '31'.

The 'destination id' is derived from the logical destination the SIP session
has been routed to. In SIP headers and Radius records there are multiple
places that contain information related to the destination. Some of them are
generated by the SIP User-Agents (hence cannot be trusted and must not be
used for accounting purposes) and others are generated by the SIP Proxy
configured by the operator and are suitable for accounting purposes.

The rating engine considers the destination to be the first non-empty CDR
field in this order:

a. CanonicalURI (the destination after all lookups inside the SIP Proxy) 
b. SipTranslatedRequestURI (the Request URI as presented by the SIP UA)
c. CalledStationId (the content of the To header, used as a last resort)

The CanonicalURI is the preferred because is reliable information generated
by the operator based on the logic configured in the SIP Proxy and the
subscriber cannot control it. OpenSIPS must be configured to send the
CanonicalURI Radius attribute when creating the radius accounting START
record. Instructions for this are available in INSTALL.txt file.

The 'destination id' is then calculated based on the longest match of this
destination field in the billing_destinations table.

The longest match for the chosen destination field is performed by the
E164_class, which by default uses E164_Europe that defines an European
numbering plan. It assumes the destinations start with a zero for a national
call and with double zero for an international call. See cdr_generic.php for
the actual logic.

If your dialing plan is different, you must use other provided class like
E164_US or create a custom class and point to it in global.inc for each
datasource as follows:

'E164_class'=>'MyE164Class',

For example the pre-defined E164_US class from cdr_generic.php matches the
American dialing plan.

Destinations are uploaded into destination table. Each reseller_id can have
its own list of destinations. If the reseller_id of trusted peers and domain
tables have been set, then the rating engine will apply the rating
information belonging to them.

Each destination has also a set of properties that can be used to alter the
calculation of the price. These properties are:

- increment: the call duration is rounded to the next increment in seconds,
  this is only used for rating, the actual CDR duration is not modified
- min_duration: set the duration to min_duration if the CDR duration is
  smaller than min_duration, this is only used for rating, the actual CDR
  duration is not modified
- max_duration: limit the duration to max_duration if CDR duration is
  bigger than max_duration, this is only used for rating, the actual CDR
  duration is not modified
- max_price: the price is set to max_price per call if the calculated price
  is bigger than the max_price


3. Determination of the costs
-----------------------------

The following steps are taken to determine the cost for the calls based on
the 'destination id' and the billing party determined at the previous steps.

1. Lookup the billing profiles in cdrtool.billing_customers table in the
   following order: subscriber,domain,gateway (based on $this->dayofweek):
   (as explained in "Determination of the billing party" above)

   - profile_name1 matches week days [1-5]
   - profile_name1_alt matches week days [1-5] if no rates for profile_name1
     are found
   - profile_name2 matches week-ends [6-0]
   - profile_name2_alt matches week-ends [6-0] if no rates for profile_name2
     are found
   - profile_name2 matches also holidays from billing_holidays table
   
   The week starts with 0 (Sunday) and ends with 6 (Saturday)

   This step determines which rates should be applied based on the day of
   the week when the call started.

2. Using the profile_name found, lookup the rate_name based
   on $this->hourofday in cdrtool.billing_profiles table

   If no rate_name is found for the given profiles a second set of profiles
   are used, profile_name1_alt and profile_name2_alt.

   - the day may be split in maximum 4 periods
   - the days starts with hour 0 and ends with hour 24
   - rate_name1 defines the first interval after hour 0
   - rate_name2 defines the first interval after rate_name1
   - rate_name3 defines the first interval after rate_name2
   - rate_name4 defines the first interval after rate_name3

   When the hour matches an interval use the rate_nameX found to lookup the
   rate in billing_rates, if no record is found use the rate called
   'default'

   This step determines which rate should be applied for the time of day
   when the call started.

3. Lookup in the cdrtool.billing_rates table the record having same name
   found at point 2 having billing_rates.destination = 'destination id'
   and billing_rates.application = application type found in the steps above.

   - return an array with all the rating values and the duration rated

   No rate will be returned if no 'destination id' is found. Make sure each
   possible destination has a 'corresponding id' and name in the destinations
   table.

   This step determines the costs within the current time span associated
   with the time of day and destination id. If the call duration exceeds
   this time span (that is a new interval for which another rate applies is
   reached), step 4 is performed.

4. If the duration rated at point 3 is less than total call duration, apply
   point 3 again for the remaining call duration in the next profile. A
   maximum of 10 spans (different rates depending of time of day, day of the
   week) can be calculated using this mechanism. After 10 spans, the engine
   bails out to avoid loops caused by invalid tables provisioning.

5. Calculate the total call Price based on its duration and connection fees.

   In global.inc there are several variables that affect how the price is
   calculated. These settings are global per CDRTool installation but some
   can be overwritten with per customer values in the billing_customers
   table.

   $RatingEngine=array( 
           "priceDenominator"        => 10000, // Rates units (global setting)
           "priceDecimalDigits"      => 4,     // Decimal information (global setting)
           "minimumDurationCharged"  => 0,     // Rate a minimum of X seconds (per customer)
           "minimumDuration"         => 0,     // Minimum duration to rate, if call duration is shorter the price is zero (per customer)
           "durationPeriodRated"     => 60     // Rate is per 60 seconds (global setting)
           "trafficSizeRated"        => 1024,  // Default we rate per 1 MB (global setting)
           "reportMissingRates"      => 0      // Send emails to administrator in case of missing rates
            );

   Pricing formula
   ---------------

   if min_duration then
        minimumDurationCharged = min_duration
   else if minimumDurationCharged set in global inc
        use minimumDurationCharged from global.inc
   else
        minimumDurationCharged = call duration

   if increment then
        durationForRating = round to the next increment
   else
        durationForRating = call duration

   if durationForRating >= minimumDurationCharged then
        Price = connectCost/priceDenominator+
        durationRate*durationForRating/durationPeriodRated/priceDenominator
   else 
        Price = 0


   ENUM discounts
   --------------

   The rating engine can apply a discount associated with the ENUM top
   level domain that returned the final destination.

   Price = Price - Price * ENUM discount / 100

   To apply ENUM based discounts, the ENUM TLD must be saved with each CDR
   and the TLDs with their corespondent discounts must be provisioned in the
   Rating tables section. See ENUM TLD discounts section for more
   information.


   Purchasing price
   ----------------

   A second price called 'Price in' is calculated using the same formula but
   based on connectCostIn and durationRateIn values. It can be used to match
   the purchasing price and calculate the margin between purchasing and
   selling prices. The information about both prices is stored in the
   RateInfo field of the CDR.  The values for connectCostIn and
   durationRateIn must be provisioned in the billing_rates and
   billing_rates_history tables using the web interface or by importing csv
   files.


7. Save the calculated Price, billing party and 'destination id' for each
   call in the CDR table. Having the price stored in the database, it is
   possible to build statistics to display consolidated revenues per country
   code, network or subscriber.


Testing the rating engine
-------------------------

You can test the rating engine by telneting to the IP and port configured in
global.inc. Type 'help' once connected to see the available commands. Use
'ShowPrice' command to simulate the rating of one session, for example:

adigeo@w1:/var/www/CDRTool/doc$ telnet ws1 9024
Trying 10.0.0.1 ...
Connected to 10.0.0.1.
Escape character is '^]'.

ShowPrice From=sip:123@example.com To=sip:0031650222333@example.com Gateway=10.0.0.1 Duration=59
0.2023
Duration: 59 s
         App: audio
 Destination: 31650
    Customer: domain=example.com
     Connect: 0.0450
   StartTime: 2009-01-03 14:29:10
--
        Span: 1
    Duration: 59 s
   ProfileId: 442 / weekend
      RateId: 442 / 0-24h
        Rate: 0.1600 / 60 s
       Price: 0.1573


Below is a description of the fields that must be separated by one or more
spaces:

 * From - must contain the caller party
 * Gateway - must contain the source IP of the session
 * To - must contain the full CanonicalURI destination, its format must be synced
   with the E164 class logic used to determine the 'destination id'
 * Duration - the duration of the session in seconds

Check the syslog for any errors, most the of the configuration errors like
missing rates are logged to the syslog.

        
Importing and exporting of rating files
---------------------------------------

It is important to set the impersonate field corectly for the login
accounts. To access in the CDRTool Rating web page all the rates table for
all resellers in the system you must set the impersonate field of the login
account to 0.0

There are different data files needed for rating. The data files are
imported into their corresponding MySQL tables. 

The files must be uploaded to /var/spool/cdrtool directory. To load the
files into the database run the following command:

/var/www/CDRTool/scripts/importRatingTables.php

Data partitioning for multiple resellers

The reseller_id column present in all rating tables is used to filter access
based on Login account impersonate field.

You may create numerical sub-directories under /var/spool/cdrtool directory.
When importing a file from such numerical directory, the reseller_id field
in all imported records is set to the directory name.

The import script knows to import the files only once so you may dump
several files over time with the same name and safely run the import script
from cron. The import script detects whether each file have been imported by
building a unique key out of the filename and the hash of the file content.
So you may use the same filenames as long as the content differs and
viceversa. If the import file has changed any records, the rating engine is
automatically instructed to reload the changes.

Sample csv files are found in the setup directory. The CSV field order is
described in setup/*.csv sample files. The first element on each line
specifies the operation will be performed with the current record. The
operation can be 2 (update/insert), 1 (insert) or 3 (delete).

The updates are performed based on a unique key present in each table:

billing_customers     - cust_idx      (reseller,gateway,domain,subscriber)
destinations          - cust_dest_idx (reseller,gateway,domain,subscriber,dest_id)
billing_profiles      - profile_idx   (reseller,name)
billing_rates         - rate_idx      (reseller,name,destination,application)
billing_rates_history - rate_idx      (reseller,name,destination,application,startDate,endDate)

The content of the rating tables can be exported in the Rating tables page.

The import script detects the type of file to import based on its filename.

The filename must comply with the following naming convention:

1. Must start with the name of the table without the billing_
2. May optionally contain extra characters after the name
3. Must end with .csv extension

Examples:

- rates.csv or rates20061201.cvs will be loaded into the rates table
- profiles.csv or profiles20061201.cvs will be loaded into the profiles table
- destinations200601.csv will be loaded in the destinations table
- ratesHistory200801.csv will be loaded in the rates_history table

Do not use 'billing_' prefix in front of the file name.

It is advisable to name the files in a consistent manner like tableYYYYMMDD.csv

The results of the import operation is logged in the database and can be
viewed in the Log section of the web interface and the syslog.


MySQL schema
------------

To see the rating tables and their structures connect to the cdrtool
database using mysql client.

Run 'show tables' and 'describe table_name':
 
billing_customers
+-------------------+
| Field             |
+-------------------+
| id                |
| reseller_id       |
| gateway           |
| domain            |
| subscriber        |
| profile_name1     |
| profile_name1_alt |
| profile_name2     |
| profile_name2_alt |
| timezone          |
+-------------------+

billing_profiles
+------------+
| Field      |
+------------+
| id         |
| reseller_id|
| name       |
| rate_name1 |
| hour1      |
| rate_name2 |
| hour2      |
| rate_name3 |
| hour3      |
| rate_name4 |
| hour4      |
+------------+

billing_rates
+-----------------+
| Field           |
+-----------------+
| id              |
| reseller_id     |
| name            |
| destination     |
| application     |
| connectCost     |
| durationRate    |
| connectCostIn   |
| durationRateIn  |
+-----------------+

billing_rates_history
+-----------------+
| Field           |
+-----------------+
| id              |
| reseller_id     |
| name            |
| destination     |
| application     |
| connectCost     |
| durationRate    |
| connectCostIn   |
| durationRateIn  |
| startDate       |
| endDate         |
+-----------------+

destinations
+--------------+
| Field        |
+--------------+
| id           |
| reseller_id  |
| gateway      |
| domain       |
| subscriber   |
| dest_id      |
| dest_name    |
| increment    |
| min_duration |
| max_duration |
| max_price    |
+--------------+

billing_holidays
+-------+
| Field |
+-------+
| day   |
+-------+

prepaid
+------------------+
| Field            |
+------------------+
| id               |
| reseller_id      |
| account          |
| domain           |    
| balance          |
| change_date      |
| active_sessions  |
| session_counter  |
+------------------+

prepaid_history
+-------------+
| Field       |
+-------------+
| id          | 
| reseller_id |
| username    | 
| domain      | 
| action      | 
| number      | 
| value       | 
| balance     | 
| date        | 
+-------------+

billing_enum_tlds
+-------------+
| Field       |
+-------------+
| id          | 
| reseller_id |
| gateway     | 
| domain      | 
| subscriber  | 
| enum_tld    | 
| e164_regexp | 
| discount    | 
+-------------+


Web based rating tables management
----------------------------------

The rating tables can be edited from the web, click on Rating tables link.
One may insert/update/delete records or apply changes on selections. For
example it is possible to increase with XX units the rate for a specific
destination.

Numeric fields support mathematical operators [+-*/], one may update
using absolute or relative values the fields in the rating tables.

The rates may be copied in bulk and start quickly working with a fresh
new rating table. Select in the Rates PSTN table by filtering on rate
name. A new button appears which allows the copy of all selected rates
into a new set. The rates are copied under the old rate id with _N
suffix where N is the next available number for which same rate id does
not exist.

The content of the rating tables can be exported into comma separated
files. The CSV format has the same structure as the import file, is
fairly easy to modify an exported batch file into an external
application and load it back into CDRTool.

Note

When $RatingEngine['split_rating_table'] is true, after changing the rates
in the web interface or by importing them, you must run the script
scripts/splitRatingTables.php to split the central billing_rates table into
individual tables for each rate id. You do not need this feature if you have
less than 100K rates in your system.


ENUM TLD discounts
------------------

To apply discounts based on ENUM certain conditions must be met.

1. The username part of the result of the ENUM lookup must be numeric and
   contain a fully qualified E164 number, optional with a numeric prefix.

   Example:

   The user dialed 020800001, the SIP Proxy has normalized the destination
   based on local policy by stripping 0 and adding country code 31 to obtain
   the fully qualified E164 number 3120800001, than it performed an ENUM
   lookup under top level domain e164.example.com for +3120800001. The ENUM
   server responsable for e164.example.com returned a response with the
   destination sip:01131208000011@gateway.example.com

   In the CDR, ENUMtld is stored as e164.example.com

   The rating logic checks if the TLD exists in the billing_enum_tlds table. 
   If it does, the rating engine tries to match the regexp field against the
   username part of the destination from the ENUM response, which has been
   saved in the Canonical URI. The match must return a fully qualified E164
   number otherwise the call is considered to have ended to a no E164
   destination and the call will be free of charge.

2. In the ENUM tld table you must provision (for the example above):

    - TLD: e164.example.com
    - Regexp: 011([1-9][0-9]{7,})
    - Discount: 25

   The parenthesis of the Regexp field indicate the E164 number returned by
   the match and discount is a percentage that will be subtracted form the
   total price of the call. The formula is described in the PSTN rating
   section.


Reloading rating tables
-----------------------

The rating engine loads some of the rating tables in the memory, when the
tables change a reload is needed. Reload of rating tables is possible
without stopping the daemon by connecting to it and issuing the reload
command. The init.d script can also be used for reloading the rating engine
with the current values from the rating database.

There are 3 ways of reloading the rating tables:

a. Each change executed in the WEB interface for rate management may update
the rating tables. If there is a change made to the database that requires a
reload the link 'Reload rating tables' appears in red color on web page.
Click on the link to execute the reload.

b. Telnet to the IP address and port number specified for the Rating engine
in global.inc. Type help to see the list of commands available. Locate the
reload rates command and execute it followed by \n. You may see the result
of the command in syslog. The results displayed by syslog will show how many
entries have been reloaded from the rating table.

c. Run /etc/init.d/cdrtool reload command


Troubleshooting
---------------

To examine the rate information for a rated call click on the Id field on
the leftmost column. (Java script support in browser is required). A blue
area will open under the CDR line containing more information about the SIP
session.

If you see no price in the CDR or no rating information appears in the call
details it means that either no destination was found in the destinations
table or no rate has been associated with that destination. Make sure that
for each entry in the destinations table there is a corespondent entry in
the rates table. CDRTool rating engine can send warning emails if it finds
missing entries in the rating tables if the system where CDRTool runs is
properly configured to send emails and the e-mail notification addresses are
set in global.inc:
 
$CDRTool['provider']['toEmail']  = "support@example.com";
$CDRTool['provider']['bccEmail'] = "cdrtool@example.com";

To log to syslog about missing rates or incorrect setup of the rating tables
enable 'reportMissingRates' in $CDRTool['rating'] section of global.inc


Renormalizing CDRs and historical rating
----------------------------------------

Sometime is useful to be able to change the rates for calls that have been
already normalized and rated, for example after changing the rating tables
you wish to apply the changes for the previous month for a customer.

To re-rate the CDRs do the following:

1. Change the current rates by using cvs files/WEB interface or add rates
   valid for specific dates/destinations in the rates_history table

2. Re-normalize the calls to be re-rated by either selecting ReNormalize
   check-box in the search screen or by changing the Normalized field in the
   CDR MySQL table (e.g. radacct):

   Examples: 

   a) Re-rate calls for this month (2004-12) SIP domain example.com:    
   UPDATE radacct set Normalized = '0' where Realm = 'example.com'
   and AcctStartTime >= '2004-12-01' 

   b) Re-rate calls for SIP subscriber sip01@example.com:       
   UPDATE radacct set Normalized = '0' where UserName = 'sip01@example.com'

3. Apply rating again using command: 

   /var/www/CDRTool/scripts/normalize.php

Notes

Renormalization process can take long time during which your database
(radacct table) will be intermitently locked. Perform this operation
only during low traffic periods. 

It is advisable to re-rate only the CDRs for destinations that have
different rates. To do this, select a filter in the CDR search screen, if
the selection is right re-run the query by selecting Re-normalize button.

After renormalization, the monthly usage information used by the quota
system will be out of date. At the next run of the quotaCheck script, a full
table scan will be performed. See QuotaSystem.txt for more information about
quota.

Holidays must be added as individual days YYYY-MM-DD in table
billing_holidays. The profile applied for holidays is the same as for
week-ends. Holidays are global and cannot be specified per customer.

Renormalization process does not affect the balance of prepaid users.
Prepaid is a real time un-reversible process, it goes in one direction. The
prepaid balance is changed only by placing a call or adding credit to it.
There are several reasons for this:

- The balance before and after each CDR is not known to be able to roll it
  back at a later time
- Re-rating correctly is mathematically not possible for prepaid users that
  have calls in progress
- If the prices are higher than previously debited and end up with a
  negative balance, the software cannot force the user to pay more
  retroactively

Re-normalization for the purpose of re-rating is useful only for postpaid
accounts where you send an invoice at the end of the month and your can
change things back and forth. If you need to perform manual credit/debit
operations to some prepaid users because of faulty pricing, you can edit in
CDRTool in the prepaid table the balance by using + or -.


Known limitations
-----------------

The rating engine does not calculate prices based on the outbound carriers
or outbound gateways, the rating plan is is assigned by the calling party
and not by called party.

Price discounts (except those based on ENUM tld) must be applied outside
CDRTool, in the billing system that prints the actual invoices. CDRTool has
no possibility to rate only calls after X minutes per month for subscriber
Y, all calls are rated uniformly.


Performance
-----------

Rating is part of the normalization process that happens every time a query
is executed in the web interface or when the rating engine is contacted by
the SIP Proxy or by the User Agent that performs the prepaid application.

The following tests have been performed between two machines with 3 GHz CPU
and 1 GB memory located on the same LAN having a round trip time of 0.2 ms.

The rating tables have been populated durring the tests with:

Aug 11 11:25:43 sip03 CDRTool[4945]: Loaded 8135 destinations
Aug 11 11:25:43 sip03 CDRTool[4945]: Loaded 6 profilesPSTN
Aug 11 11:25:43 sip03 CDRTool[4945]: Loaded 4 holidays
Aug 11 11:25:44 sip03 CDRTool[4945]: Loaded 7273 prepaid accounts

Tests results for the postpaid application:

Clients         Rating command                  Execution speed
----------------------------------------------------------------
1               ShowPrice                       390/s per client 
5               ShowPrice                       100/s per client
10              ShowPrice                       60/s per client


Tests results for the prepaid application

Clients         Rating command                  Execution speed
----------------------------------------------------------------
1               MaxSessionTime/DebitBalance     250/s per client
5               MaxSessionTime/DebitBalance     80/s per client
10              MaxSessionTime/DebitBalance     40/s per client

Client means either a SIP Proxy entity or a CDRTool server, which performs
the normalization process.

The rates are read directly from MySQL from version 6.1 instead of
being cached as they are found based on a known index.

To monitor the connections to the rating engin server telnet to
the rating engine port and issue the ShowClients command. Example output:

ShowClients

Clients:

1. 91.20.228.143:32837
2. 85.1.86.71:57945
3. 91.20.228.146:35098
4. 91.20.228.150:34285
5. 91.20.228.129:55090
6. 91.20.228.164:34147

Requests:

12 requests from 91.20.228.129
11 requests from 91.20.228.150
8 requests from 85.11.86.71
2 requests from 91.20.228.164

Statistics:

Total requests: 33
Uptime: 169 seconds
Load: 0.20/s