Given a manifold with boundary endowed with an action of a discrete group on it, we consider the algebra of operators generated by elements in the Boutet de Monvel algebra of pseudodifferential boundary value problems and shift operators acting on functions on the manifold and its boundary. Provided that the group is of polynomial growth and its action is isometric, we construct a Chern character for elliptic elements in this algebra with values in a de Rham type cohomology of the fixed point manifolds for the group action and obtain an index formula in terms of this Chern character. Our index formula contains as special cases the index formula by Fedosov for boundary value problems in the Boutet de Monvel algebra and the index formula by Nazaikinskii, Savin and Sternin for operators on a closed manifold associated with an isometric group action. © 2021, The Author(s), under exclusive licence to Springer Nature Switzerland AG.