Monitoring & Evaluation (M&E)  

Introduction

 

Overall Coordination of M&E Activities

 

            Project monitoring and evaluation (M&E) will be carried out as three separate but distinct efforts.  First, concurrent monitoring will be the responsibility of the National Coordinator (O&M) assisted by a Project M&E Consultant (who will be charged with responsibilities of the day-to-day M&E operations).  Second, project monitoring and evaluation cells (PME cells) or CMUs will regularly monitor and report on the sub-project’s physical and financial inputs and outputs, at the consortia level. Third, an independent entity will be charged to carry out comprehensive outcome-focused impact evaluations of the NAIP at three stages: baseline, second MTR and project completion.

 

            The institutional structure of the M&E process will follow the overall organizational and governance structure of the NAIP. The primary responsibility for M&E will be with the PIU, led by the ND and assisted by the national coordinators (NCs), and the finance and procurement officers. The PIU will report to the NSC and the PMC on M&E issues and will inform the O&MPC and the RPC. The monitoring effort will lend support to: management in its day-to-day operations; developing a project monitoring and tracking system (PMTS); development of information networks; provide support to the overall NAIP communication dissemination effort; business development and planning; learning and capacity building; and consortium activities in developing concept notes into full proposals.  The M&E system will identify problem areas at national and consortium level, help management to administer project processes, aid project design and implementation, help make mid-course corrections, and help the project achieve its overall development

 

Institutional Structure for M&E

 

At the National level

 

            The primary responsibility for monitoring, evaluation and reporting will rest with the National Coordinator (O&M) assisted by the PME Consultant. The M&E Consultant’s first charge will be the design of a PMTS for NAIP implementation within the overall management framework for the NAIP. The M&E Consultant will also be responsible for assisting the NSC, PMC, PIU and to help coordinate the M&E related information needs of the four NCs.  Specific other responsibilities of the M&E Consultant include:

 

• Conducting the benchmark survey for the NAIP as a whole.

• Coordinate with the PIU staff reporting processes.

• Consolidate reports and contribute to the overall NAIP reporting requirements.

• Provide guidance to M&E activities of the consortia.

• Contribute to the further development of MISs in the NAIP.

• Ensure gender-oriented monitoring at national and consortium level.

 

At the Consortium level

 

            A Consortium Monitoring Unit (CMU) will be established in each funded consortium. The CMU will report directly to the CL. The work program of the CMU will be developed in consultation with the NC 1 at the PIU and approved and cleared by the chairperson of the CAC.

 

The CMU will be responsible for (but not be restricted to):

 

• Preparing an M&E work plan and budget.

• Preparing half yearly and annual reports.

• Undertaking regular field trips to introduce and document key M&E practices.

• Preparing and submitting half yearly and annual monitoring and supervision reports.

• Planning and developing PME related training programs;.

• Designing and conducting M&E exposure sessions.

• Assisting the consortium leader in all matters relating to M&E planning.

• Supporting and assisting in the World Bank and NAIP supervision visits.

 

M&E of Outcomes and Results

 

            The NAIP will develop a results-based M&E system that will monitor project processes using the following methods and tools:

 

• A well-defined Results Framework that is derived from clearly defined goals, objectives, outputs and activities with corresponding indicators, means of verification and key assumptions;
• A well-defined M&E strategy regarding information requirements, tools and methodologies for data collection, analysis and reporting;
• A comprehensive M&E plan with clear roles and responsibilities with respect to data gathering and reporting; and
• Internal and External periodic assessment and evaluations which include base-line studies, beneficiary assessments, mid-term evaluations, ex-post evaluations and impact evaluations.

 

M&E and Performance Indicators:  Monitoring indicators will be consolidated by the PIU and used to update the key performance indicators for the project.  These benchmarks would be developed during the base-line study and further refined as necessary by the PIU and CACs during project implementation.  Field level implementation would be independently monitored by CMUs, in consultation with the PIU, by: (a) intensive regular periodic monitoring; (b) regular visits to project sites.  As part of the work plan, the CMUs would also be engaged in impact evaluation studies, including beneficiary assessments, under the guidance of NCAP.  In the disadvantaged areas, farmers, farmer organizations and NGOs assisting with implementation of the project would also be involved with M&E activities.  Private sector participation in consortium activities would also be monitored by the CMUs.

 

A well-equipped and suitably trained central unit at the national level will be responsible for guiding the overall M&E effort within the project and vis-à-vis partners, plus providing timely and relevant information to the NAIP management team, stakeholders and project partners. This will require close coordination and communication with those in-charge of M&E in the consortia, the NSC and the ND, external consultants and field staff where appropriate, as well as members of external M&E missions.

 

Each consortium will be required to establish its own M&E capacity and to design its own M&E plans. This will be laid out in the “Guidelines for Consortium Development”. Issues that will be addressed by all consortium M&E groups are: consortium formation and management; research plan preparation and implementation; knowledge management; capacity building; product development and technology transfer; financial management; procurement; and impact evaluation. In addition, consortia can add further indicators as required by their own unique nature. The CMU will report to the CL and the CAC.

 

Institutional Arrangements.   The operational responsibility for planning and coordinating M&E activities would rest with the PIU, with one NC responsible for overseeing this activity.  The PIU would utilize a computerized project tracking system to monitor progress and to link with monitoring activities within the participating entities at the national level as well as at the consortium level.  At the consortium level, the PIU would be assisted by the CMUs.  

 

Reporting Arrangements.   The PIU would submit to the Bank: (a) up-to-date physical and financial expenditure data compared to annual and end-project targets; (b) updated indicators of project performance compared to annual and end of project targets; (c) successes and problems encountered during the reporting period with suggested remedial actions; and (d) social and environmental impacts of the project 

 

Half-yearly assessment of the progress for each consortium will be undertaken by the CAC. The responsible NC will consolidate the reports from the CACs in the component under his supervision and send them to the concerned the TAG, which in turn will submit these to the RPC. A Peer Review Team comprising external experts will subject each project at the consortium level to an MTR and an evaluation at the end of the project.

 

The CMUs would submit to the PIU: (i) quarterly financial and procurement reports summarizing concurrent monitoring observations; (ii) six-monthly reports summarizing project M&E of preceding six months, cross-cutting issues and recommendations, and updated project indicators and; (iii) three comprehensive reports - the base-line survey and two consortium evaluations (at the project MTR and project completion).

 

Half-Yearly Reports:  Half-yearly reports would be prepared by CLs and submitted to the CAC. Once approved, the CAC would then transmit this report to the PIU and make it available on the NAIP websites.  Under the guidance of the NC1, the M&E consultant would prepare a consolidated report to include the four components. The final compiled version including the financial and audit reports will be made available to the RPC and to the World Bank (Appendix 15).

 

Annual Reports: Draft annual reports would be prepared by consortia in the following manner. The CAC would appoint a Project Review Team that may include a representative or designate from the PIU. The PRT would then prepare a report and submit it to the CL and the Chairman of the CAC.  The CAC would then organize an annual workshop to discuss the report to be attended by the PRT, and members of the TAG. Afterwards the CL will prepare the annual report and submit this through the CAC to the PIU. For each component the responsible NC would then compile an overall annual report. At the same time the M&E consultant under the guidance of the NC1 would prepare a report that covers all the NAIP operations (Appendix 16).  

 

            At the release of the overall annual reports, an annual workshop will be organized.  This will be a two-day event. The first day’s program will consist of concurrent component discussions. Participants at this event will include the CAC Chairmen, TAG (for Components 2, 3 and 4) or O&MAG (for Component 1) members, the NCs, and the CLs.  The second day will consist of two separate sessions. Session 1 will focus on technical issues while session two will deal with administrative issues. The technical sessions will cover M&E operations, work program issues and reporting effectiveness. Based on this session and the sessions of the day before, a comprehensive annual report will be compiled by the ND and submitted to the PMC for review and comments. The report will then be made available on relevant web sites (ICAR/SAU) and submitted to the World Bank. The workshops for the second, fourth and last years of the project will coincide with the first and second mid-term and, end-of-project review missions of the World Bank, respectively.

 

Mid-term Reports:    The mid-term report includes the following but is not limited to:

 

• Trends towards socio-economic impact.
• Production aspects.
• Environmental and social aspects.
• Lessons learnt, mid-course corrections and redirection of sub-projects.

 

Final Report of the Consortium: A completion report at the end of the sub-project should bring out the results and achievements of the research work and its impact on the science, commerce, society and socio-economic conditions of the farmers. The innovations in technology introduced as a result of implementation of the scheme should be brought out. The impact of HRD in decentralization of research management should also be highlighted. The report should also delineate the trends whether the improvement/ stagnation/ reduction in productivity, profitability, sustainability, environmental benefits and HRD (capacity) varied in relation to bench-mark status. The write-up should also highlight important milestones achieved and success stories during the sub-project period and send one set of publications to the PIU. The report should also track the sources whether the findings were used and contributed to better decision-making, saving of the resources, etc. The proforma prescribed for submission of the final report is annexed (Appendix 17). The report must be submitted within two months of the completion of the sub-project. All reports shall be examined for completeness, attachment of relevant documents in support of achievements claims and the projected expenditure plan. Participating institutions and/ or PIs may be asked to provide clarifications explanations and make amend­ments/ modifications.  Complete data in electronic form should be passed onto the PIU-NAIP.

 

For the component 1, in case of the Business Development Plan sub-project where there will be a consortium, the same procedure for half-yearly and the annual reports will be followed. For other sub-projects the O&MAG will take the role of the CACs. The O&M PC will take the role of the RPC.

 

 Implementation Arrangements for M&E of Components 2, 3, and 4

 

Output Indicators.  M&E will focus on the measurement of outputs and outcomes and on the quality of the management processes in the consortia. Some of the important output categories concern:

 

·        Improved crop varieties/animal breeds/tree species-both number and adoption rate

·        Improved management practices – both numbers and adoption rate;

·        Publications and reports – Variety and numbers;

·        Training events – Variety and number;

·        Dissemination (technology transfer) events – Variety and number;

·        Public services – Variety and number;

·        Professional recognitions – Variety and number; and

·        Product development – Variety and number.

 

 Outcome Indicators: Research impact can only be assessed by establishing a causal relationship between outputs and benefits to the end users (outcome). These include:

 

·        Area expansion (crops) and/or increase in number (animals);

·        Increase in productivity;

·        Reduction in production costs;

·        Enhanced profitability to the producers and other participants in the value chain;

·        Sustainability of adoption (technology);

·        Enhanced gainful employment;

·        Income generation/enhancement;

·        Improvement in food consumption; and

·        Improvement in access to credit.

 

 Management Process Indicators.  Management processes will be monitored through the annual reports of the consortium. Focus will be on decision making in the CAC, the extent to which the consortium retains or strengthen its inclusiveness, the internal procedures for overcoming differences of opinion and conflict management, and the quality and timeliness of financial management and procurement.

 

Data Collection and System Development Efforts.  Consortium members would assume the primary responsibility for collecting data to update the input/output indicators.  The data would be consolidated and managed by the PIU.  The CMUs would assist the PIU to develop a computerized plan for the project. For the concurrent monitoring and impact evaluation studies, the CMUs would be primarily responsible for data collection and collation, using independent surveys, and for report preparation.  Preparation of evaluation studies for different components would be an ongoing process culminating in one major interim review of the project (just prior to the second MTR) and a final project completion review.   A base-line survey would be carried out by an M&E consultant in coordination with the implementing entities.  The survey would be completed and a draft report prepared within six months of project effectiveness. Questionnaires and formats for this survey should be prepared within one month of the award of the M&E contract, pre-tested in the field, and cleared with the PIU prior to any actual data collection efforts.

 

An on-line Project Monitoring & Tracking System (PMTS) will be developed to collect and process M&E-related information. The proposed system will be used to facilitate the capture, storage and retrieval of a clear, quantified and operational base-line data; to track the progress of various projects under components 2, 3 and 4 on an on-going basis; to monitor the progress of the overall NAIP project; to regularly assess the performance of project staff; and to evaluate the output and outcome at mid-term and prior to completion. The PMTS would be developed in-house and will include modules for outreach and dissemination information, activity profiles, monitoring and evaluation and performance measurement.

 

            A scorecard will be developed for the purpose of comparing the performance of the consortia on an annual basis. The scorecard results will be made available on the NAIP website.

 

M&E Process

           Each consortium will go through roughly three phases: (i) an initial phase of six months for pre-project activities in which the focus will be on needs analysis, orientation and sensitization of people, and identification of targets and suitable output and outcome indicators; (ii) project implementation per se; and (iii) a final phase of six months for post-project activities in which reports containing information on outputs, outcomes, dissemination, and success are brought out; and in which planning for follow-up activities (continuation/ further expansion/commercialization) is finalized.

 

Table : Results Framework

PDO	Outcome Indicators	Use of Outcome Information
The specific objective is to accelerate the collaborative development and application of agricultural innovations between public research organizations, farmers, private sector and other stakeholders	·       The number of partnerships between public research system, private sector and other stakeholders ·       Increase in agricultural innovations by end of the project	·      PY2: Make adjustments to policy reform agenda ·      PY3: Address capacity building issues ·      PY4-5: Determine success of project and make recommendations for future interventions
Component 1: ICAR as the catalyzing agent for management of change in the Indian NARS	·       A substantial increase in the number of knowledge products and public awareness messages that the system shares with others ·       Business development units will be established in five ICAR institutes/State Agricultural Universities (SAUs) ·       The ICAR system has an enhanced capacity for policy analysis, planning and advice ·       ICAR has enhanced financial and procurement management capacity	·       PY2:  Review decision making structures while addressing component strategy ·       PY 3:  Identify newer models that would increase sharing information across the network ·      PY 4 – 5:  Address capacity issues and provide guidance on research strategies, market linkages
Component 2: Research on production to consumption systems	·      Number of technologies that have been jointly developed between consortia partners ·      Improved quality management mechanisms in the selected production to consumption systems ·      Number of public-private partnerships developed	·       PY1-2: Determine progress by consortia for improvement of value chains, strategy and interventions. ·      PY3: Determine if collaboration amongst institutions can be increased
Component 3: Research on sustainable rural livelihood security	·      Number of technologies developed  and tested in interaction with target groups ·      Increased rural employment opportunities ·      The number of farmer organizations engaged in defining and implementing collaborative research	·      PY1: Determine progress and address issues to overcome slow progress ·      Determine if  interaction with target groups is effective ·      PY2:  Determine if strategy needs to address inadequacy of supply and services.
Component 4: Basic and strategic research in the frontier areas of agricultural science	·       Number of publications of Indian agricultural scientists in high quality journals ·       Number of patent applications	·      PY2:  Determine progress to address issues related to scientific publishing. ·      PY2-3:  Determine if IP management strategy needs to be re adjusted

Table 2: Arrangements for Results Monitoring- Outcomes and Outputs, Component 1

 

 

 

Table 3: Arrangements for Results Monitoring- Outcomes and Outputs, Component 2