# Bulk Disbursement Integration ### Bulk Payment Design * Provide scalable and fault tolerant design with easy integration * Allow easy plugging with different payment types (Mojaloop, GSMA, Afrimoney and others) * Enable reuse of existing BPMNs to facilitate transfers #### Prioritizing New Use Cases * G2P * P2G #### Bulk Processor Flow ![](https://1303323730-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-L_J6oxfkTjM9Go4YNSW%2Fuploads%2FNLR0YxoisBDrInLU2bio%2Fbulk_processor.png?alt=media\&token=af22d79f-e037-4e17-8b6d-46c46d3f4d14) * Bulk processing is designed as a separate system within Payment Hub EE architecture supporting * File integrity check with MD5/SHA256 checksum when received in channel connector * Bulk processor * Split streamed files and split into sub-batches * Format file contents into a format bulk connectors can parse * Order payments * Publish to Kafka payment topic * Merge back sub-file and sub-batch results into a consolidated response * Bulk connector * Format data based on payment connector requirements * Starting point of initiated payment type-specific workflows * Stream sub-batch transactions from S3 based on metadata from Kafka #### Configurations To provide a granular level of control, the bulk processor supports multiple configurations for different stages like ordering, formatting, splitting etc. * partylookup.enable: used to enable or disable the partylookup stage * approval.enable: used to enable or disable the approval stage * ordering.enable: used to enable or disable the ordering of transactions in a data set based on a field provided. * ordering.field: the field based on which ordering will be done. * formatting.enable: enable or disable the formatting of data set. If formatting is disabled the default formatting standard is used. * formatting.standard: the formatting standard to be used for transactions, possible values are "DEFAULT", "GSMA". * splitting.enable: enable or disable the splitting of the original csv file into sub-batches. Splitting is done on the sub-batch-size field configuration. * splitting.sub-batch-size: the size of the single sub batch. * mergeback.enable: enable or disable the merging back of the sub batches result. * success-threshold-check.enable: enable or disable the success threshold check, use this configuration to make sure at least x percentage of transaction is successful in a batch. * success-threshold-check.success-rate: a percentage value that will be used while making the threshold check. * acknowledgement.enable: enable or disable the acknowledgement part of the workflow. **Key Considerations:** * Kafka provides auto partitioning within topics based on throughput and data * Scalable and fault-tolerant system provided by the multi-cluster setup * Handle backpressure from Zeebe and throttle workflow initiation * Kafka will act as a buffer in this case * Advantageous when Zeebe is overloaded * Due to resource limitations * Or not fast enough scaling * Reduces cascading Denial of Service for other payment connectors. * Provide a visual workflow representation with Zeebe * Stores every state for workflow instances * Timers surviving node failures without interruptions * Using an efficient, high-performance binary protocol (gRPC) for the clients to connect and receive tasks for execution * Retry mechanism on the entire subprocess * But retry is already in place within subprocesses * Unified payment architecture * Easy integration with AMS if payment staging requires it * In case of beneficiary creation during transfer * Using Raft for log replication * Kafka to write ahead log replication as well. * Easy integration of pre and post-transaction workers * Such as bulk notification #### References: ## API Specification * Provide developer friendly consumer APIs * Enable easy integration by abstracting underlying payment type API complexities * Group common fields across various payment APIs and make em mandatory field * All fields in the spec below are mandatory fields * Allow addition of payment type specific fields as optional ### 1. Bulk Transfer Use below csv file for refernece. ```csv id,request_id,payment_mode,payer_identifier_type,payer_identifier,payee_identifier_type,payee_identifier,amount,currency,note,program_shortcode,cycle 0,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,slcb,accountNumber,003001003874120160,accountNumber,927549027483,850,USD,Test Payee Payment,MDM,1 1,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,slcb,accountNumber,003001003873110196,accountNumber,927549027483,222,USD,Test Payee Payment,MDM,1 2,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,slcb,accountNumber,003001003874120160,accountNumber,927549027483,850,USD,Test Payee Payment,MDM,1 3,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,slcb,accountNumber,003001003873110196,accountNumber,927549027483,222,USD,Test Payee Payment,MDM,1 4,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,mojaloop,accountNumber,003001003874120160,mosip,437892675228,850,USD,Test Payee Payment,COV,1 5,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,mojaloop,accountNumber,003001003873110196,mosip,437892675228,222,USD,Test Payee Payment,COV,1 6,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,mojaloop,accountNumber,003001003874120160,mosip,437892675228,850,USD,Test Payee Payment,COV,1 7,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,mojaloop,accountNumber,003001003873110196,mosip,437892675228,222,USD,Test Payee Payment,COV,1 8,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,gsma,accountNumber,003001003874120160,msisdn,8837461856,850,USD,Test Payee Payment,CWP,2 9,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,gsma,accountNumber,003001003873110196,msisdn,8837461856,222,USD,Test Payee Payment,CWP,2 10,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,gsma,accountNumber,003001003874120160,msisdn,8837461856,850,USD,Test Payee Payment,CWP,2 11,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,gsma,accountNumber,003001003873110196,msisdn,8837461856,222,USD,Test Payee Payment,CWP,2 ``` #### Use the below request along with above CSV file {% code overflow="wrap" %} ```bash curl --location --request POST 'https://bulk-connector.sandbox.fynarfin.io/batchtransactions?type=csv' \ --header 'Platform-TenantId: ibank-usa' \ --header 'X-CorrelationID: 3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8' \ --header 'filename: ph-ee-bulk-demo-6.csv --form 'data=@"ph-ee-bulk-demo-6.csv"' ``` {% endcode %} #### Use the below request to start a batch only using the filename {% code overflow="wrap" %} ```bash curl --location --request POST 'https://bulk-connector.sandbox.fynarfin.io/batchtransactions?type=raw' \ --header 'Platform-TenantId: ibank-usa' \ --header 'X-CorrelationID: 3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8' \ --header 'filename: ph-ee-bulk-demo-6.csv' ``` {% endcode %} #### Use the below request to pass transaction data in the body of the request ```bash curl --location --request POST 'https://bulk-connector.sandbox.fynarfin.io/batchtransactions?type=raw' \ --header 'Platform-TenantId: ibank-usa' \ --header 'X-CorrelationID: 3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8' \ --header 'Content-Type: application/json' \ --data-raw '[ { "creditParty": [ { "key": "msisdn", "value": "8837461856" } ], "debitParty": [ { "key": "accountnumber", "value": "003001003874120160" } ], "subType": "SLCB", "amount": "820.00", "currency": "RWF", "descriptionText": "Test Payment" } ]' ``` ```json { "batch_id": "738bd830-3bd9-4511-bb42-3d0f5798e014", "request_id": "3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8", "status": "queued" } ``` ### 2. Bulk Transfer Status {% code overflow="wrap" %} ```bash curl --location --request GET 'https://ops-bk.sandbox.fynarfin.io/api/v1/batchtransactions/3112a0ba-0733-4133-ae24-fc3310cb7dfe' \ --header 'Platform-TenantId: ibank-usa' \ --header 'Authorization: Bearer token' ``` {% endcode %} 
{  
  "batch_id": "UUID",
  "request_id": "UUID",
  "notes": "Bulk transfers",
  "status": "processing",
  "mode": "{payment_mode}",
  "purpose": "refund",
  "total": 1000,
  "successful": 980,
  "failed": 20,
  "file": "S3 link", 
  "created_at": 1545383037,
  "totalAmount": "1050000.00",
  "successfulAmount": "975000.00",
  "failedAmount": "25000.00",
  "pendingAmount": "50000.00",
  "currency": "SLE"
}
#### Response File ```csv id,request_id,payment_mode,payer_identifier_type,payer_identifier,payee_identifier_type,payee_identifier,amount,currency,note,program_shortcode,cycle,status,error_code,error_description 0,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,slcb,accountNumber,003001003874120160,accountNumber,927549027483,850,USD,Test Payee Payment,MDM,1,SUCCESS,, 1,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,slcb,accountNumber,003001003873110196,accountNumber,927549027483,222,USD,Test Payee Payment,MDM,1,SUCCESS,, 2,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,slcb,accountNumber,003001003874120160,accountNumber,927549027483,850,USD,Test Payee Payment,MDM,1,SUCCESS,, 3,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,slcb,accountNumber,003001003873110196,accountNumber,927549027483,222,USD,Test Payee Payment,MDM,1,FAILURE,422,Inactive Account 4,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,mojaloop,accountNumber,003001003874120160,mosip,437892675228,850,USD,Test Payee Payment,COV,1,SUCCESS,, 5,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,mojaloop,accountNumber,003001003873110196,mosip,437892675228,222,USD,Test Payee Payment,COV,1,SUCCESS,, 6,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,mojaloop,accountNumber,003001003874120160,mosip,437892675228,850,USD,Test Payee Payment,COV,1,SUCCESS,, 7,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,mojaloop,accountNumber,003001003873110196,mosip,437892675228,222,USD,Test Payee Payment,COV,1,PENDING,, 8,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,gsma,accountNumber,003001003874120160,msisdn,8837461856,850,USD,Test Payee Payment,CWP,2,SUCCESS,, 9,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,gsma,accountNumber,003001003873110196,msisdn,8837461856,222,USD,Test Payee Payment,CWP,2,SUCCESS,, 10,f1e22fe3-9740-4fba-97b6-78f43bfa7f2f,gsma,accountNumber,003001003874120160,msisdn,8837461856,850,USD,Test Payee Payment,CWP,2,PENDING,, 11,72aa3ea4-e6f6-4880-877f-39f6ac4d052e,gsma,accountNumber,003001003873110196,msisdn,8837461856,222,USD,Test Payee Payment,CWP,2,SUCCESS,, ``` ### 3. Batch Details {% code overflow="wrap" %} ```bash curl --location --request GET 'https://ops-bk.sandbox.fynarfin.io/api/v1/batchtransactions/45e2baca-b087-4d90-8392-da2961f9b9ed/detail?pageNo=1&pageSize=10' \ --header 'Platform-TenantId: ibank-usa' \ --header 'Authorization: Bearer Token' ``` {% endcode %} ```json [ { "id": 568, "workflowInstanceKey": 2251799907439003, "transactionId": "e5eea064-1445-4d32-bc55-bd9826c779a0", "startedAt": 1629130966000, "completedAt": 1629130967000, "status": "IN_PROGRESS", "statusDetail": null, "payeeDfspId": "giraffe-bank", "payeePartyId": "7854278651", "payeePartyIdType": "MSISDN", "payeeFee": null, "payeeFeeCurrency": null, "payeeQuoteCode": null, "payerDfspId": "gorilla-bank", "payerPartyId": "7543010", "payerPartyIdType": "MSISDN", "payerFee": null, "payerFeeCurrency": null, "payerQuoteCode": null, "amount": 448, "currency": "USD", "direction": "OUTGOING", "errorInformation": "{\\\"errorInformation\\\":{\\\"errorDescription\\\":\\\"Invalid responseCode 500 for transfer on PAYER side, transactionId: e5eea064-1445-4d32-bc55-bd9826c779a0 Message: {\\\\\\\"timestamp\\\\\\\":1629130966891,\\\\\\\"status\\\\\\\":500,\\\\\\\"error\\\\\\\":\\\\\\\"Internal Server Error\\\\\\\",\\\\\\\"message\\\\\\\":\\\\\\\"\\\\\\\",\\\\\\\"path\\\\\\\":\\\\\\\"/fineract-provider/api/v1/interoperation/transfers\\\\\\\"}\\\",\\\"errorCode\\\":\\\"4101\\\"}}", "batchId": "c02a14f0-5e7e-44a1-88eb-5584a21e6f28" } ] ``` ### 4. Get Batches {% code overflow="wrap" %} ```bash curl --location --request GET 'http://ops-bk.sandbox.fynarfin.io/api/v1/batches?page=2&size=10&sortedBy=requestFile&sortedOrder=asc' \ --header 'Platform-TenantId: ibank-usa' ``` {% endcode %} {% code overflow="wrap" %} ```json { "content": [ { "id": 57, "batchId": "6bdd4e23-9357-481b-b814-b678b146a01b", "subBatchId": null, "requestId": "3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8", "requestFile": "1661791795354_ph-ee-bulk-demo-6.csv", "totalTransactions": 11, "total": 12, "ongoing": 0, "failed": 3, "successful": 9, "totalAmount": 6432, "successfulAmount": 4510, "pendingAmount": 0, "failedAmount": 1922, "result_file": "https://paymenthub-ee-dev.s3.us-east-2.amazonaws.com/1663238730228_6bdd4e23-9357-481b-b814-b678b146a01b.csv", "resultGeneratedAt": null, "note": null, "workflowKey": null, "workflowInstanceKey": 2251799813689497, "startedAt": 1661791796000, "completedAt": 1661791796000 }, { "id": 29, "batchId": "837f0b1a-d06c-4a7b-a6bb-a274e7e3cf43", "subBatchId": null, "requestId": "1aff3859d4e8-fab3a4d5-0f4f-4e78-b6b5", "requestFile": "1661793332698_ph-ee-bulk-demo-6.csv", "totalTransactions": 12, "ongoing": 0, "failed": 12, "completed": 0, "totalAmount": 3810, "ongoingAmount": null, "failedAmount": 3810, "completedAmount": null, "result_file": "https://paymenthub-ee-dev.s3.us-east-2.amazonaws.com/5618434551624_39484844071623_response.csv", "resultGeneratedAt": 1661844072000, "note": null, "workflowKey": null, "workflowInstanceKey": 2251799813692279, "startedAt": 1661793333000, "completedAt": 1661793333000 }, { "id": 33, "batchId": "de2b9caf-07f0-43d0-ab98-34e070f52913", "subBatchId": null, "requestId": "1", "requestFile": "1661798671312_1", "totalTransactions": 50, "ongoing": null, "failed": null, "completed": 50, "totalAmount": 12490, "ongoingAmount": null, "failedAmount": null, "completedAmount": 12490, "result_file": "https://paymenthub-ee-dev.s3.us-east-2.amazonaws.com/1661873857624_1661843746423_response.csv", "resultGeneratedAt": null, "note": null, "workflowKey": null, "workflowInstanceKey": 2251799813699494, "startedAt": 1661798674000, "completedAt": null }, { "id": 34, "batchId": "eb94687b-5eee-4bc9-ad8a-e89bf0604dc1", "subBatchId": null, "requestId": "3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8", "requestFile": "1661840850289_ph-ee-bulk-demo-6.csv", "totalTransactions": null, "ongoing": null, "failed": null, "completed": null, "totalAmount": null, "ongoingAmount": null, "failedAmount": null, "completedAmount": null, "result_file": "https://paymenthub-ee-dev.s3.us-east-2.amazonaws.com/1661849863624_1661983851623_response.csv", "resultGeneratedAt": null, "note": null, "workflowKey": null, "workflowInstanceKey": 2251799813750207, "startedAt": 1661840852000, "completedAt": 1661840853000 }, { "id": 38, "batchId": "7f2a7366-3cc1-4293-8cc7-21f862872f7b", "subBatchId": null, "requestId": "3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8", "requestFile": "1661840967146_ph-ee-bulk-demo-6.csv", "totalTransactions": null, "ongoing": null, "failed": null, "completed": null, "totalAmount": null, "ongoingAmount": null, "failedAmount": null, "completedAmount": null, "result_file": "https://paymenthub-ee-dev.s3.us-east-2.amazonaws.com/1661883987873_1661844071623_response.csv", "resultGeneratedAt": null, "note": null, "workflowKey": null, "workflowInstanceKey": 2251799813750554, "startedAt": 1661840967000, "completedAt": 1661840968000 }, { "id": 42, "batchId": "a7159479-e0d8-4a77-8f92-8b903b3173b7", "subBatchId": null, "requestId": "3a4dfab5-0f4f-4e78-b6b5-1aff3859d4e8", "requestFile": "1661843665396_ph-ee-bulk-demo-6.csv", "totalTransactions": 12, "ongoing": 0, "failed": 12, "completed": 0, "totalAmount": null, "ongoingAmount": null, "failedAmount": null, "completedAmount": null, "result_file": "https://paymenthub-ee-dev.s3.us-east-2.amazonaws.com/1661844087873_1661844087873_response.csv", "resultGeneratedAt": 1661844088000, "note": null, "workflowKey": null, "workflowInstanceKey": 2251799813754434, "startedAt": 1661843667000, "completedAt": 1661843668000 } ], "totalElements": 79, "totalPages": 8, "last": false, "numberOfElements": 10, "sort": [ { "direction": "ASC", "property": "requestFile", "ignoreCase": false, "nullHandling": "NATIVE", "descending": false, "ascending": true } ], "first": false, "size": 10, "number": 2 } ``` {% endcode %} ### 5. Download Template {% code overflow="wrap" %} ```bash curl --location --request GET 'https://bulk-connector.sandbox.fynarfin.io/template?types=PROGRAM,MOJALOOP' ``` {% endcode %} {% code title="Template.csv" %} ```csv id,request_id,payment_mode,payer_identifier_type,payer_identifier,payee_identifier_type,payee_identifier,amount,currency,note,program_shortcode,cycle ``` {% endcode %} ## Payment Routing * Route enables user to split payments received and transfer the funds to various vendors/beneficiaries. * Sample request body ```json { "amount": 2000, "currency": "INR", "transfers": [ { "account": "acc_CPRsN1LkFccllA", "amount": 1000, "currency": "INR", "notes": { "branch": "Acme Corp Bangalore North", "name": "Gaurav Kumar" }, }, { "account": "acc_CNo3jSI8OkFJJJ", "amount": 1000, "currency": "INR", "notes": { "branch": "Acme Corp Bangalore South", "name": "Saurav Kumar" }, } ] } ``` * With Auto Payout * Targeting individuals * Money is transferred directly to bank accounts / mobile money wallets from the sender float account. * No involvement of AMS. * Without Auto Payout * Targeting organisations * Money is transferred to Fineract wallets of the receivers. * Receiver can choose if they want to withdraw the money out of the wallet * Or consider using the money in float/current/checking account for L2 beneficiaries (clear invoices, payroll) seamlessly in one platform (L1 beneficiary amount remains in closed loop) * This provides an advantage to beneficiary orgs to manage their payouts if they have a legacy current/checking account with mostly manual payout processing.