Bulk Disbursement Integration

PreviousDevelopment Environment NextRelease Notes

Last updated 1 year ago

Was this helpful?

Bulk Disbursement Integration

Bulk Payment Design

Provide scalable and fault tolerant design with easy integration
Allow easy plugging with different payment types (Mojaloop, GSMA, Afrimoney and others)
Enable reuse of existing BPMNs to facilitate transfers

Prioritizing New Use Cases

Bulk Processor Flow

Bulk processing is designed as a separate system within Payment Hub EE architecture supporting
File integrity check with MD5/SHA256 checksum when received in channel connector
Bulk processor
- Split streamed files and split into sub-batches
- Format file contents into a format bulk connectors can parse
- Order payments
- Publish to Kafka payment topic
- Merge back sub-file and sub-batch results into a consolidated response
Bulk connector
- Format data based on payment connector requirements
- Starting point of initiated payment type-specific workflows
- Stream sub-batch transactions from S3 based on metadata from Kafka

Configurations

To provide a granular level of control, the bulk processor supports multiple configurations for different stages like ordering, formatting, splitting etc.

partylookup.enable: used to enable or disable the partylookup stage
approval.enable: used to enable or disable the approval stage
ordering.enable: used to enable or disable the ordering of transactions in a data set based on a field provided.
ordering.field: the field based on which ordering will be done.
formatting.enable: enable or disable the formatting of data set. If formatting is disabled the default formatting standard is used.
formatting.standard: the formatting standard to be used for transactions, possible values are "DEFAULT", "GSMA".
splitting.enable: enable or disable the splitting of the original csv file into sub-batches. Splitting is done on the sub-batch-size field configuration.
splitting.sub-batch-size: the size of the single sub batch.
mergeback.enable: enable or disable the merging back of the sub batches result.
success-threshold-check.enable: enable or disable the success threshold check, use this configuration to make sure at least x percentage of transaction is successful in a batch.
success-threshold-check.success-rate: a percentage value that will be used while making the threshold check.
acknowledgement.enable: enable or disable the acknowledgement part of the workflow.

Key Considerations:

Kafka provides auto partitioning within topics based on throughput and data
Scalable and fault-tolerant system provided by the multi-cluster setup
Handle backpressure from Zeebe and throttle workflow initiation
- Kafka will act as a buffer in this case
Advantageous when Zeebe is overloaded
- Due to resource limitations
- Or not fast enough scaling
Reduces cascading Denial of Service for other payment connectors.
Provide a visual workflow representation with Zeebe
Stores every state for workflow instances
Timers surviving node failures without interruptions
Using an efficient, high-performance binary protocol (gRPC) for the clients to connect and receive tasks for execution
Retry mechanism on the entire subprocess
- But retry is already in place within subprocesses
Unified payment architecture
Easy integration with AMS if payment staging requires it
- In case of beneficiary creation during transfer
Using Raft for log replication
- Kafka to write ahead log replication as well.
Easy integration of pre and post-transaction workers
- Such as bulk notification

References:

https://forum.camunda.io/t/zeebe-usecase-architecture-question/1168/6

https://stage.docs.zeebe.io/reference/grpc.html#error-handling

https://docs.camunda.io/docs/product-manuals/zeebe/deployment-guide/operations/backpressure/

https://stage.docs.zeebe.io/bpmn-workflows/multi-instance/multi-instance.html

https://camunda.com/blog/2019/08/zeebe-horizontal-scalability/

https://docs.camunda.io/docs/0.25/product-manuals/zeebe/bpmn-workflows/embedded-subprocesses/embedded-subprocesses/

https://docs.camunda.io/docs/0.25/product-manuals/zeebe/bpmn-workflows/event-subprocesses/event-subprocesses

https://medium.com/@jayphelps/backpressure-explained-the-flow-of-data-through-software-2350b3e77ce7

API Specification

Provide developer friendly consumer APIs
Enable easy integration by abstracting underlying payment type API complexities
Group common fields across various payment APIs and make em mandatory field
- All fields in the spec below are mandatory fields
Allow addition of payment type specific fields as optional

1. Bulk Transfer

Use below csv file for refernece.

id,request_id,payment_mode,payer_identifier_type,payer_identifier,payee_identifier_type,payee_identifier,amount,currency,note,program_shortcode,cycle
0,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,slcb,accountNumber,003001003874120160,accountNumber,927549027483,850,USD,Test Payee Payment,MDM,1
1,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,slcb,accountNumber,003001003873110196,accountNumber,927549027483,222,USD,Test Payee Payment,MDM,1
2,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,slcb,accountNumber,003001003874120160,accountNumber,927549027483,850,USD,Test Payee Payment,MDM,1
3,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,slcb,accountNumber,003001003873110196,accountNumber,927549027483,222,USD,Test Payee Payment,MDM,1
4,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,mojaloop,accountNumber,003001003874120160,mosip,437892675228,850,USD,Test Payee Payment,COV,1
5,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,mojaloop,accountNumber,003001003873110196,mosip,437892675228,222,USD,Test Payee Payment,COV,1
6,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,mojaloop,accountNumber,003001003874120160,mosip,437892675228,850,USD,Test Payee Payment,COV,1
7,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,mojaloop,accountNumber,003001003873110196,mosip,437892675228,222,USD,Test Payee Payment,COV,1
8,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,gsma,accountNumber,003001003874120160,msisdn,8837461856,850,USD,Test Payee Payment,CWP,2
9,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,gsma,accountNumber,003001003873110196,msisdn,8837461856,222,USD,Test Payee Payment,CWP,2
10,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,gsma,accountNumber,003001003874120160,msisdn,8837461856,850,USD,Test Payee Payment,CWP,2
11,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,gsma,accountNumber,003001003873110196,msisdn,8837461856,222,USD,Test Payee Payment,CWP,2

Use the below request along with above CSV file

curl --location --request POST 'https://bulk-connector.sandbox.fynarfin.io/batchtransactions?type=csv' \
--header 'Platform-TenantId: ibank-usa' \
--header 'X-CorrelationID: 3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8' \
--header 'filename: ph-ee-bulk-demo-6.csv
--form 'data=@"ph-ee-bulk-demo-6.csv"'

Use the below request to start a batch only using the filename

curl --location --request POST 'https://bulk-connector.sandbox.fynarfin.io/batchtransactions?type=raw' \
--header 'Platform-TenantId: ibank-usa' \
--header 'X-CorrelationID: 3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8' \
--header 'filename: ph-ee-bulk-demo-6.csv'

Use the below request to pass transaction data in the body of the request

curl --location --request POST 'https://bulk-connector.sandbox.fynarfin.io/batchtransactions?type=raw' \
--header 'Platform-TenantId: ibank-usa' \
--header 'X-CorrelationID: 3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8' \
--header 'Content-Type: application/json' \
--data-raw '[
    {
        "creditParty": [
            {
                "key": "msisdn",
                "value": "8837461856"
            }
        ],
        "debitParty": [
            {
                "key": "accountnumber",
                "value": "003001003874120160"
            }
        ],
        "subType": "SLCB",
        "amount": "820.00",
        "currency": "RWF",
        "descriptionText": "Test Payment"
    }
]'

{
    "batch_id": "738bd830-3bd9-4511-bb42-3d0f5798e014",
    "request_id": "3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8",
    "status": "queued"
}

2. Bulk Transfer Status

curl --location --request GET 'https://ops-bk.sandbox.fynarfin.io/api/v1/batchtransactions/3112a0ba-0733-4133-ae24-fc3310cb7dfe' \
--header 'Platform-TenantId: ibank-usa' \
--header 'Authorization: Bearer token'

{  
  "batch_id": "UUID",
  "request_id": "UUID",
  "notes": "Bulk transfers",
  "status": "processing",
  "mode": "{payment_mode}",
  "purpose": "refund",
  "total": 1000,
  "successful": 980,
  "failed": 20,
  "file": "S3 link", 
  "created_at": 1545383037,
  "totalAmount": "1050000.00",
  "successfulAmount": "975000.00",
  "failedAmount": "25000.00",
  "pendingAmount": "50000.00",
  "currency": "SLE"
}

Response File

id,request_id,payment_mode,payer_identifier_type,payer_identifier,payee_identifier_type,payee_identifier,amount,currency,note,program_shortcode,cycle,status,error_code,error_description
0,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,slcb,accountNumber,003001003874120160,accountNumber,927549027483,850,USD,Test Payee Payment,MDM,1,SUCCESS,,
1,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,slcb,accountNumber,003001003873110196,accountNumber,927549027483,222,USD,Test Payee Payment,MDM,1,SUCCESS,,
2,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,slcb,accountNumber,003001003874120160,accountNumber,927549027483,850,USD,Test Payee Payment,MDM,1,SUCCESS,,
3,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,slcb,accountNumber,003001003873110196,accountNumber,927549027483,222,USD,Test Payee Payment,MDM,1,FAILURE,422,Inactive Account
4,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,mojaloop,accountNumber,003001003874120160,mosip,437892675228,850,USD,Test Payee Payment,COV,1,SUCCESS,,
5,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,mojaloop,accountNumber,003001003873110196,mosip,437892675228,222,USD,Test Payee Payment,COV,1,SUCCESS,,
6,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,mojaloop,accountNumber,003001003874120160,mosip,437892675228,850,USD,Test Payee Payment,COV,1,SUCCESS,,
7,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,mojaloop,accountNumber,003001003873110196,mosip,437892675228,222,USD,Test Payee Payment,COV,1,PENDING,,
8,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,gsma,accountNumber,003001003874120160,msisdn,8837461856,850,USD,Test Payee Payment,CWP,2,SUCCESS,,
9,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,gsma,accountNumber,003001003873110196,msisdn,8837461856,222,USD,Test Payee Payment,CWP,2,SUCCESS,,
10,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,gsma,accountNumber,003001003874120160,msisdn,8837461856,850,USD,Test Payee Payment,CWP,2,PENDING,,
11,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,gsma,accountNumber,003001003873110196,msisdn,8837461856,222,USD,Test Payee Payment,CWP,2,SUCCESS,,

3. Batch Details

curl --location --request GET 'https://ops-bk.sandbox.fynarfin.io/api/v1/batchtransactions/45e2baca-b087-4d90-8392-da2961f9b9ed/detail?pageNo=1&pageSize=10' \
--header 'Platform-TenantId: ibank-usa' \
--header 'Authorization: Bearer Token'

[
    {
        "id": 568,
        "workflowInstanceKey": 2251799907439003,
        "transactionId": "e5eea064-1445-4d32-bc55-bd9826c779a0",
        "startedAt": 1629130966000,
        "completedAt": 1629130967000,
        "status": "IN_PROGRESS",
        "statusDetail": null,
        "payeeDfspId": "giraffe-bank",
        "payeePartyId": "7854278651",
        "payeePartyIdType": "MSISDN",
        "payeeFee": null,
        "payeeFeeCurrency": null,
        "payeeQuoteCode": null,
        "payerDfspId": "gorilla-bank",
        "payerPartyId": "7543010",
        "payerPartyIdType": "MSISDN",
        "payerFee": null,
        "payerFeeCurrency": null,
        "payerQuoteCode": null,
        "amount": 448,
        "currency": "USD",
        "direction": "OUTGOING",
        "errorInformation": "{\\\"errorInformation\\\":{\\\"errorDescription\\\":\\\"Invalid responseCode 500 for transfer on PAYER side, transactionId: e5eea064-1445-4d32-bc55-bd9826c779a0 Message: {\\\\\\\"timestamp\\\\\\\":1629130966891,\\\\\\\"status\\\\\\\":500,\\\\\\\"error\\\\\\\":\\\\\\\"Internal Server Error\\\\\\\",\\\\\\\"message\\\\\\\":\\\\\\\"\\\\\\\",\\\\\\\"path\\\\\\\":\\\\\\\"/fineract-provider/api/v1/interoperation/transfers\\\\\\\"}\\\",\\\"errorCode\\\":\\\"4101\\\"}}",
        "batchId": "c02a14f0-5e7e-44a1-88eb-5584a21e6f28"
    }
]

4. Get Batches

curl --location --request GET 'http://ops-bk.sandbox.fynarfin.io/api/v1/batches?page=2&size=10&sortedBy=requestFile&sortedOrder=asc' \
--header 'Platform-TenantId: ibank-usa'

{
    "content": [
        {
            "id": 57,
            "batchId": "6bdd4e23-9357-481b-b814-b678b146a01b",
            "subBatchId": null,
            "requestId": "3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8",
            "requestFile": "1661791795354_ph-ee-bulk-demo-6.csv",
            "totalTransactions": 11,
            "total": 12,
            "ongoing": 0,
            "failed": 3,
            "successful": 9,
            "totalAmount": 6432,
            "successfulAmount": 4510,
            "pendingAmount": 0,
            "failedAmount": 1922,
            "result_file": "https://paymenthub-ee-dev.s3.us-east-2.amazonaws.com/1663238730228_6bdd4e23-9357-481b-b814-b678b146a01b.csv",
            "resultGeneratedAt": null,
            "note": null,
            "workflowKey": null,
            "workflowInstanceKey": 2251799813689497,
            "startedAt": 1661791796000,
            "completedAt": 1661791796000
        },
        {
            "id": 29,
            "batchId": "837f0b1a-d06c-4a7b-a6bb-a274e7e3cf43",
            "subBatchId": null,
            "requestId": "1aff3859d4e8-fab3a4d5-0f4f-4e78-b6b5",
            "requestFile": "1661793332698_ph-ee-bulk-demo-6.csv",
            "totalTransactions": 12,
            "ongoing": 0,
            "failed": 12,
            "completed": 0,
            "totalAmount": 3810,
            "ongoingAmount": null,
            "failedAmount": 3810,
            "completedAmount": null,
            "result_file": "https://paymenthub-ee-dev.s3.us-east-2.amazonaws.com/5618434551624_39484844071623_response.csv",
            "resultGeneratedAt": 1661844072000,
            "note": null,
            "workflowKey": null,
            "workflowInstanceKey": 2251799813692279,
            "startedAt": 1661793333000,
            "completedAt": 1661793333000
        },
        {
            "id": 33,
            "batchId": "de2b9caf-07f0-43d0-ab98-34e070f52913",
            "subBatchId": null,
            "requestId": "1",
            "requestFile": "1661798671312_1",
            "totalTransactions": 50,
            "ongoing": null,
            "failed": null,
            "completed": 50,
            "totalAmount": 12490,
            "ongoingAmount": null,
            "failedAmount": null,
            "completedAmount": 12490,
            "result_file": "https://paymenthub-ee-dev.s3.us-east-2.amazonaws.com/1661873857624_1661843746423_response.csv",
            "resultGeneratedAt": null,
            "note": null,
            "workflowKey": null,
            "workflowInstanceKey": 2251799813699494,
            "startedAt": 1661798674000,
            "completedAt": null
        },
        {
            "id": 34,
            "batchId": "eb94687b-5eee-4bc9-ad8a-e89bf0604dc1",
            "subBatchId": null,
            "requestId": "3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8",
            "requestFile": "1661840850289_ph-ee-bulk-demo-6.csv",
            "totalTransactions": null,
            "ongoing": null,
            "failed": null,
            "completed": null,
            "totalAmount": null,
            "ongoingAmount": null,
            "failedAmount": null,
            "completedAmount": null,
            "result_file": "https://paymenthub-ee-dev.s3.us-east-2.amazonaws.com/1661849863624_1661983851623_response.csv",
            "resultGeneratedAt": null,
            "note": null,
            "workflowKey": null,
            "workflowInstanceKey": 2251799813750207,
            "startedAt": 1661840852000,
            "completedAt": 1661840853000
        },
        {
            "id": 38,
            "batchId": "7f2a7366-3cc1-4293-8cc7-21f862872f7b",
            "subBatchId": null,
            "requestId": "3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8",
            "requestFile": "1661840967146_ph-ee-bulk-demo-6.csv",
            "totalTransactions": null,
            "ongoing": null,
            "failed": null,
            "completed": null,
            "totalAmount": null,
            "ongoingAmount": null,
            "failedAmount": null,
            "completedAmount": null,
            "result_file": "https://paymenthub-ee-dev.s3.us-east-2.amazonaws.com/1661883987873_1661844071623_response.csv",
            "resultGeneratedAt": null,
            "note": null,
            "workflowKey": null,
            "workflowInstanceKey": 2251799813750554,
            "startedAt": 1661840967000,
            "completedAt": 1661840968000
        },
        {
            "id": 42,
            "batchId": "a7159479-e0d8-4a77-8f92-8b903b3173b7",
            "subBatchId": null,
            "requestId": "3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8",
            "requestFile": "1661843665396_ph-ee-bulk-demo-6.csv",
            "totalTransactions": 12,
            "ongoing": 0,
            "failed": 12,
            "completed": 0,
            "totalAmount": null,
            "ongoingAmount": null,
            "failedAmount": null,
            "completedAmount": null,
            "result_file": "https://paymenthub-ee-dev.s3.us-east-2.amazonaws.com/1661844087873_1661844087873_response.csv",
            "resultGeneratedAt": 1661844088000,
            "note": null,
            "workflowKey": null,
            "workflowInstanceKey": 2251799813754434,
            "startedAt": 1661843667000,
            "completedAt": 1661843668000
        }
    ],
    "totalElements": 79,
    "totalPages": 8,
    "last": false,
    "numberOfElements": 10,
    "sort": [
        {
            "direction": "ASC",
            "property": "requestFile",
            "ignoreCase": false,
            "nullHandling": "NATIVE",
            "descending": false,
            "ascending": true
        }
    ],
    "first": false,
    "size": 10,
    "number": 2
}

5. Download Template

curl --location --request GET 'https://bulk-connector.sandbox.fynarfin.io/template?types=PROGRAM,MOJALOOP'

Template.csv

id,request_id,payment_mode,payer_identifier_type,payer_identifier,payee_identifier_type,payee_identifier,amount,currency,note,program_shortcode,cycle

Payment Routing

Route enables user to split payments received and transfer the funds to various vendors/beneficiaries.
Sample request body

{
  "amount": 2000,
  "currency": "INR",
  "transfers": [
    {
      "account": "acc_CPRsN1LkFccllA",
      "amount": 1000,
      "currency": "INR",
      "notes": {
        "branch": "Acme Corp Bangalore North",
        "name": "Gaurav Kumar"
      },
    },
    {
      "account": "acc_CNo3jSI8OkFJJJ",
      "amount": 1000,
      "currency": "INR",
      "notes": {
        "branch": "Acme Corp Bangalore South",
        "name": "Saurav Kumar"
      },
    }
  ]
}

With Auto Payout
- Targeting individuals
- Money is transferred directly to bank accounts / mobile money wallets from the sender float account.
- No involvement of AMS.
Without Auto Payout
- Targeting organisations
- Money is transferred to Fineract wallets of the receivers.
- Receiver can choose if they want to withdraw the money out of the wallet
- Or consider using the money in float/current/checking account for L2 beneficiaries (clear invoices, payroll) seamlessly in one platform (L1 beneficiary amount remains in closed loop)
- This provides an advantage to beneficiary orgs to manage their payouts if they have a legacy current/checking account with mostly manual payout processing.