Error in problem setup for large problems in python

I am trying to use osqp for minimization of a large problem with many constraints in python and I get this error:

File “./main.py”, line 59, in step1
prob.setup(P, q, A, lb, ub,warm_start=False)
File “/usr/local/lib/python3.8/site-packages/osqp/interface.py”, line 32, in setup
self._model.setup(*unpacked_data, **settings)
ValueError: Workspace allocation error!

I noticed that my machines memory usage goes to 1.2 GB and then stops there. Is there a way to work around this error?

I works fine for the same code (for building the matrices and constraints) but smaller case.

Thank you!

What is the dimension of your problem and the number of nonzeros in P and A? It could just be that the matrix that OSQP wants to factor at the start has way too many nonzeros in it.

Also - which linear solver are you using in OSQP? It is very likely to be either QDLDL or the Pardiso from MKL. Maybe try switching from one to the other?

Thanks for your response. The sizes of P and A are as follows:
P -> size: 4010400x4010400, nnz=12017832
A -> size: 4120800x4010400, nnz=11556000

I have tried both solvers and I get the same error.

The problem is very likely to be that the factors of your KKT matrix have way too many nonzeros to fit in memory. The KKT matrix looks structurally like:

K = [P+I,  A'
      A,  -I],

although the identity matrices will be scaled up or down. All of the solvers supported in the standard v0.6 distribution of OSQP are direct solvers, meaning that they compute a cholesky-like factorisation of this matrix after some permutation is applied.

The number of nonzeros in the factors is very strongly influenced by the sparsity pattern of A and P, but unless your problem is exceptionally well structured you will get at least some amount of fill-in in the KKT factors. It sounds as if the solver is not able to allocate sufficient memory for these factors.

It is possible to implement a custom linear solver for OSQP, and your problem almost certainly requires an indirect solver implementation to be solvable. You could also wait until v1.0 of OSQP, which should have support for 1-2 indirect solver methods. I can’t give an ETA for that version though.

Thanks you! I will try to implement an interface for an indirect solver.

You can try running your code on a machine with more RAM. Also, if you have a GPU that supports CUDA, you can try running our CUDA implementation of OSQP, which implements an indirect linear system solver.